Namecheap had an outage, so I guess I will now try them!

namecheap

Have you ever thought of trying a service AFTER you heard about an outage it had?

Well this is what happened to me after I read a post by Namecheap explaining, in detail, how the outage they had happened on November 13 and the steps they plan to take to ensure this does not happen again.

Now Namecheap is on my shortlist of registrars to try in the next few months as a secondary domain name registrar.

I really like it when companies are transparent about a problem they had and they take full responsibility. That is even if the problem was problem was not 100% their fault. I like it when they seem to know exactly what went wrong and they explain how they will fix it to ensure it never happens again.

I don’t like it when some companies don’t even acknowledge there was a problem and sometimes blame their customers.

Companies should also listen to their customers and it is not the first time that Namecheap does something similar.

So, I guess the geek in me found the post below really exciting!

Statement on the Namecheap Outage, November 13th

No company likes an outage. At Namecheap, we do everything in our power to maximise availability and uptime, while minimizing downtime because we understand the disruption it causes to you and your business.

What happened on November 13

Unfortunately on November 13, at approximately 5 AM UTC, we experienced a major outage caused by a power failure in our primary datacenter in Phoenix, Arizona.

For our primary datacenter, we operate out of PhoenixNAP, a top-tier datacenter renowned for reliability, security, and infrastructure resilience. We are one of the major tenants, consuming around half a Megawatt of power and hosting thousands of servers within the datacenter. We consume floor space and power from PhoenixNAP, while running our own network, hardware and onsite operations.

During the night, electrical engineers were engaged in maintenance on one of the UPS battery backup systems associated with one of the power feeds available in the datacenter itself. Maintenance on power infrastructure is routine, and happens on a weekly basis. Up until now, we’ve never had an issue with any power maintenance window.

On the November 13, however, routine power maintenance turned into the first power outage in PhoenixNAP’s 10 year history and caused one of the power feeds in the datacenter to go down several times.

The result on Namecheap’s services

For the vast majority of Namecheap’s services, power outage should never be an issue. This is because our core network, platforms, website, hosting, and cloud are all connected to redundant power circuits, taking power from an A-feed as well as a fully separated B-feed.

Unfortunately, we did experience an issue. Some of our infrastructure (non-critical areas) are not redundant by design. Elements of this infrastructure include network equipment. So when the primary power feed failed, this network equipment went down, and started rebooting and going down throughout the unstable period. This caused route flapping on our core distribution network equipment, that in turn caused us to lose connectivity to the datacenter.

One of our 4 core routers even went into kernel panic due to the unexpected load pattern. On top of the flapping problem, several of our legacy network switches and load balancers were not properly configured to sustain full A-feed outage, and therefore went down as well.

The resulting situation was that many servers and services remained up, but did not have outside network connectivity to serve customers.

Resolving the issue, and guarding against future problems

Once the A-side power was returned after the outage, the flapping stopped. This allowed us to bring all of our services back online, and restore the full Namecheap experience.

To be transparent and open, this was an error on our behalf, and one that should not have happened.

For that, we again apologize. For us, mistakes are both an opportunity to learn, as well as improve. And to further our commitment to full and open transparency, we are taking the following immediate steps to ensure this does not happen again:

  1. Introduction of network changes to ensure that, in future, the non redundant portion of network equipment does not affect the redundant portion in case of an outage—especially for cases when power feeds are unstable.
  2. An audit and fix of the legacy network equipment to ensure it is properly configured and tested for full feed failures.
  3. Improvement of our internal policies and procedures around how frequently power audits and live tests are done.

On behalf of our Executive Team, we would again like to take this opportunity to apologize for the disruption that we caused to you, our customers.

Regards,

The Namecheap Executive Team

Sold.Domains

About Konstantinos Zournas

Konstantinos studied Computer Engineering and Computer Science in London and lives in Athens, Greece. He loves domains and building websites. He is online since 1995, learned about html in 1996 and got into domains in 2002. He started the OnlineDomain.com blog in 2012.

9 comments

  1. I like NameCheap because of their prices and interface but the outage worries me.
    (Also NameCheap don’t have a way to bulk export auth codes which is frustrating)

    • Konstantinos Zournas

      All online companies have outages. Even YouTube recently had one.
      GoDaddy has small or bigger outages every once in a while. Sometimes you can’t login, sometimes domain auctions are down, etc.
      It is normal. The difference is how you communicate the outages and what you do to reduce them and become better.
      For example someone from GoDaddy may acknowledge a problem but I don’t remember a formal explanation of an outage and more importantly I don’t see them trying not to see the same thing happening again.

    • Konstantinos Zournas

      Not having a bulk export function would be a deal breaker for me.

  2. Just to add they handled it the best they could and nowadays it’s rare that the truth about these things comes out. It looks like they’ve held their hands up, honest and sincere reflection and positive change. Props to that!

  3. As a person who values customer support and business ethics, I’m glad to see NameCheap being transparent and taking full responsibility of the issue. Yet the sole reason that I haven’t registered a domain name with them is their brand name. Personally, “NameCheap” doesn’t sound professional to me. Just my two cents.

  4. I have recently started using their hosting for one of my new projects. I liked the interface and the quick support I got for a small issue.

  5. I like them but prices are higher than other places and I don’t believe they offer free forwarding?

    • They do offer free forwarding and free DNS, everyone does these days. They dont have any sort of bulk configuration (i.e. if you’d like to change name servers for multiple domains at once). They dont even have the full list of domains you could save/convert to excel/whatever if you keep bundles of domains with different registrars and want to keep track on them manually – the list is a “download after, remove before” type of page, very annoying.

  6. I dont know much about NameCheap but back to the original topic of your post here.

    I agree with you. The way they stated everything about their outage, apology, explanation, and further solutions to guarantee this will not happen again was nothing short of first class.

    Kudos to NameCheap for such stellar communication after an issue.

    Very cool of you to highlight this Konstantinos.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.