Post mortem of the overnight DDoS Attack
Overnight on October 24, 2011 between the hours of approximately 12:55am and 2:55am ET the NS1 and NS2 anycast constellations came under a DDoS Attack. The attack was a combined syn flood and DNS flood which impaired the following anycast constellations to varying degrees:
ns1.easydns.com/remote1.easydns.com (a.k.a dns1.easydns.com)
ns2.easydns.com/remote2.easydns.com (a.k.a dns2.easydns.net)
The attack primarily impacted domains on the old platform or if domains on the new platform were affected, they were only delegated to the dns1 and dns2 nameserver constellations.
Scope of Impact
It is looking like the members affected were reporting problems primarily from the East and West Coasts of North America and seems to be (thankfully) limited, either by network geography and/or the time of day of the occurrence.
What You Can Do
If you are still on the old platform, please consider migrating to the new system sooner than later. We will actually be making an announcement about this fairly soon, but this is where everybody is headed anyway. On the new system at the very least, you get more nameservers: at least one more anycast strand and between 5 and 10 additional servers worldwide (as many as 16 more for Enteprise DNS customers). Our complete nameserver deployment on the new system is outlined here – any impact reports last night from users on the new system were still using the old nameserver delegations from the old system or hadn’t added all available nameservers to their domains.
If you are on the new platform, please double check your nameserver delegations. If you are only delegated to DNS1 and DNS2, please add the appropriate additional nameservers.
- If we are your registrar, then you simply need to click on the “nameservers” link in the Domain Overview and select “Use easyDNS Nameservers”.
- If you are using an external registrar, reference this chart to see which nameservers to use for your delegation.
We We’re Doing
We have located member domain which was the target of last night’s attack and upon examination found that it was violating the easyDNS AUP (they almost always are…) We have terminated service to this domain and it has since moved off of our nameservers.
Over the years we’ve found that the best way to mitgate a DOS attack is to not be the target of one, and we screen incoming domains with an eye toward filtering out “high probability” targets. As we’ve remarked before:
Generally there are two kinds of DOS targets:
- Targets you never knew were using your system and when you find out, you want to take a shower. These are the targets 90% of the time: ponzi schemes, virus distributors, phishing sites, etc. These are usually scumbags who make a lot of enemies and are already violating your AUP. The DOS brings your attention to their presence on your system, you throw them under a bus and the DOS follows them. Problem solved.
- Then there’s “high profile” and “hot button” customers who people try to DOS just for bragging rights or some sort of vendetta. These are harder to handle, as you don’t want to cut off somebody who has a legit right to co-exist on the net with everybody else. Sometimes, you have to pull the plug, even temporarily to give yourself time to think and figure out your next move. I mean this in a very generic way: datacenters do this, network carriers do this, web hosts do this: if it gets too intense for your upstream, whammo, you’re null routed until things cool down. This is just how it works out here.
We consider last night’s target among the former. We have made additions to our signup filters that would block similar domains from coming here in the future.
Additional Nameserver Deployment
We are in the process of adding another node to the DNS1.easydns.com located at the BlackLotus datacenter in L.A. This will bring DNS1.easydns.com up to 6 nodes globally and adds some always welcome additional DOS-mitigation muscle to DNS1.
We are going to be accelerating our plans to migrate members away from the old platform and into the new one. The new easyDNS platform as more nameserver deployments and as we consolidate our member domains and have just one platform to deal with, we will have an easier time keeping it beefed up and expanding redundancy.
We are very sorry to any member who experienced issues because of this. We have striven over the years to be as redundant to DDoS attacks as possible, but as we have noted in our “DNS and DOS attacks” article (see below), even though anycast DNS deployments greatly decrease the scope of impact in a DDoS situation, it is still a major letdown for those people who are impacted. I want to personally extend apologies to those sysops out there were woken up at 1am on a Sunday night by their pagers and PDAs because their chosen nameserver vendor was in a firefight.
Monday, Oct 24 DNS resolution issues [resolved]
We are currently (as of 2:45 AM Eastern) experiencing resolution issues related to some servers in our dns1.easydns.com and dns2.easydns.net anycast clusters. These issues may be causing resolution issues for some customers. Our systems group is aware of the issue and is working hard on getting things back up to par. Updates will be made in this space as the situation evolves.
UPDATE 3:25 Eastern: The situation is now under control. All nameservers are responding normally and our systems group is monitoring the situation closely. If you do experience further issues, please let us know via firstname.lastname@example.org.
UPDATE: The issues we experienced last night were the effect of a DDOS attack on easyDNS nameservers. Throughout all of this our anycast clusters did remain online however certain nodes were flooded with traffic and became unresponsive. The attack has subsided and we are working internally and with our vendors to determine who the target was.
– easyDNS Support
Saturday, Oct 22, 2011 – No Phone Support
Due to an unexpected scheduling conflict, our support team is unfortunately not able to offer phone support today. We will still be answering emails as per usual; if you have any support requests, please send them to email@example.com
Verisign domain takedown proposal very worrisome.
Under a proposed Verisign initiative, all .COM/.NET domains exist at the pleasure of the United States government.
Verisign just released an overview of their proposed “Anti-Abuse Domain Use Policy” Under ICANN’s Registry Services Evaluation Process. The program’s chief aim is to provide a takedown mechanism of malicious websites distributing malware. In itself, not a bad thing, considering some registrars are unresponsive toward abuse or network stability issues.
However, lumped in with the conditions under which Verisign can invoke their takedown capabilities are some troubling “add ons”, as quoted below:
With the rest of the world, we are saddened to learn of Steve Jobs’ passing. Everyone at easyDNS feels a special connection to Apple and its products. We use Apple products everyday to build the websites, write the code and keep our business humming.
Steve Jobs was a creative genius who showed millions that technology should be accessible and easy to use. We remain inspired by his commitment to customer service, his passion for functional technology and his relentless pursuit of excellence in the face of adversity.
In Steve’s honor, we have made a donation to Pancreatic Cancer Canada. www.pancreaticcancercanada.ca