Go vote in the CIRA elections, if you haven't already

The voting process for the CIRA election is on for a couple more days, ending on the 29th.

As I always remark, the .CA space is quite unique in that it gives the primary stakeholders, the .CA domain registrants, unprecedented ability to shape policy via direct Board elections, yet the turnout is often fairly sparse.

One of my pet peeves with the overall structure of the Board elections has come to light with a vengeance, when I went to look at who I felt I should support this year.

I like the members’ nominated side of the ballot much more than the Nominations Committee side this year. Nothing against the NomCom, but I simply don’t know enough about many of those candidates to be able to comment intelligently on them. Yet, we can vote for up to four candidates from the NomCom side, but only one candidate from the members’ nominations.

I don’t like that, plain and simple. If you’re going to have elections, then put the candidates on the ballot and let people decide who they want. All this differentiating between “NomCom” choices and “members” choices just smells like banana republic, communist dictator jerry-rigging to me. If there’s 5 slots, then put the NomCom candidates there, put the members’ candidates there, and then let the electorate do what they’re supposed to do and sort it out with their votes.

So even though I’m supposed to pick 4 people from the Nominations Committee side, I only voted for one: Paul Andersen. He’s a friend, he’s a competitor, he’s the Chairman of the Board presently. He can always be counted on to be a voice of reason.

And even though I was only allowed to vote for one choice on the members’ candidates side, I will endorse three of them, depending on your priorities.

  1. I voted for Zak Muskovitch
  2. If you’re main concern is around technical stability issues, then Andrew Sullivan
  3. And Rick Anderson, also a long-time Board veteran, is always a good addition.

But as I always say each year, the important thing is to get out there and vote.

Ok, here's what happened (with that DNS timeout thing this morning)

This morning around 8:30am EST monitoring services around the world, things like DNSStuff, Pingdom, etc began reporting DNS timeout errors for easyDNS back to their respective users. Oddly, most of the time a human check (like trying to get to a website in a web browser) would succeed. Not always.

To compound the weirdness, none of our datacenters were reporting or seeing anything like a DOS attack, non-standard traffic or anything of the like.

To further compound the problem, the symptoms were very intermittent and sporadic, surfacing one moment in one area (say, the San Jose nameservers, which at one point we stopped announcing via our anycast announcements), and then be ok literally the next moment.

Then we noticed a customer domain giving us the mother of all dynamic DNS updates, hitting us with over 800 host updates in under a minute. When we looked closer, it seemed to be hitting us with hundreds or thousands of dynamic host updates every few minutes.

What is supposed to happen in a case like this is the system is supposed to throttle those endpoints, giving them a “TOOSOON” error for pretty well every connection request, and if it keeps up, we blackhole the IPs sending them.

For some reason, we will call for what it is: our fault, the throttling systems for dynamic DNS were not functioning properly on the new user interface system. As a result, every few minutes all of these updates were being injected into the system, the servers throughout our network would choke as they attempted to replicate the same data hundreds or thousands of times in succession, only to have to do it again a few minutes later.

This obviously will not do and we will be spending our time re-engineering this part of the system and plan to have permanent fixes committed today. We have resumed anycast announcements for San Jose and Amsterdam nameservers.

We’re extremely sorry to have kicked off your workweek/monday morning (or monday afternoon for those users in the U.K) with an event like this. It is not or proudest moment. We will work diligently to ensure a repeat performance will never occur.

If you have any questions or concerns around this issue, feel free to email (markjr [at] easydns [dot] youknowwhat) or call, 1-888-677-4741 ext 225.

– mark

Update to nameserver resolution issue

Update: 11:29 EST

None of our data centers have reported anything looking like attack traffic, DOS activity or anything other than normal DNS query traffic coming in.

We traced most of the DNS timeouts back to San Jose, which has servers in both the NS1 and NS2 anycast strands, we have removed San Jose from both anycast announcements, which seemed to improve things. We also stopped announcing Amsterdam.

I have noticed that the last few comments to the blog “we’re seeing it too” have all come from England and the UK. We’re now looking at that.

Thanks to all for their datapoints and their patience.

It appears that the issue is only affecting certain nameservers in a few locations. All the servers are fine, but it appears that there may be some network issues affecting not only easyDNS but general network traffic in some areas or across some routes. For the moment, we have removed those servers from the answer-groups, and we believe the problem to be resolved for the time being.

We are continuing to investigate what may be causing these issues, and as yet are not considering it a DOS attack.

Please note that reports from site monitoring services may have given far more negative results than users actually experienced in accessing affected sites.

We apologize for the inconvenience, and will update the blog when we have more information.

Intermittent slowdowns on some requests

Since this morning, some easyDNS servers have been responding slowly to DNS requests. This has caused intermittent timeouts in DNS response for some pages. We apologize for any inconvenience.

Our network team is currently investigating the matter and will update this blog as soon as possible.

Price Drop on Geotrust True BusinessID Wildcard SSL Certs

I had lunch yesterday with a colleague from Tucows, who is , among other things, our wholesale supplier of SSL certificates from Geotrust. I mentioned we had a customer who wanted 10 True BusinessID Wildcard SSL Certs and he said he would check on the possibility of getting us a price break on that order.

Later in the afternoon I was happy to receive the following email:

I’ll make it simple and reduce your wildcard pricing for all orders, not just for the 10 but all.

Will be effective tonight.

So we’re spreading the joy and reducing our price on the Geotrust True BusinessID Wildcard SSL Certificates from $559 to $519.

Geotrust retails these SSL certs at $995, because the big benefit of these SSL certs is that they provide up to 256-bit encryption on an unlimited number of subdomains on your webserver. These are also “full company verification” SSL certs, which means you also get a GeoTrust True Site Seal with company name and date/time stamp to put on your website.

You can see our SSL Pricing here, and to order an SSL cert simply log into your easyDNS account and click on the “SSL Certificates” item in the lefthand sidebar menu.