Tucows issues an apology regarding the Enom outage

Tucows issued an apology regarding the Enom outage. Enom had a large-scale data center migration that caused many different issues.

You can see the Tucows apology here:

Beginning Saturday, January 15, 2022, Enom experienced a series of complications with a planned data center migration that caused significant disruptions for a subset of our customers.

We sincerely apologize to all of those impacted. We pride ourselves on being a reliable domain registration platform, and this weekend we fell short. We are committed to regaining your trust and to serving you better.

A full internal audit is underway and an incident report is forthcoming. This will include a summary of events and scope, learnings, and policy and process changes to mitigate future issues.

In the meantime, we have consolidated an overview and timeline of events to transparently share with our valued community.

Sincerely,

David Woroch
EVP, Tucows Domain Services

---

Overview

Enom had a large-scale data center migration planned for Saturday, January 15, 2022, from 09:00 EST to 21:00 EST.

We first notified Enom resellers of this migration via email on December 7, 2021, with reminder emails on January 7 and January 12, 2022. In these communications, resellers were told that they would be unable to register and manage domains, email, and SSL certificates through the Enom Control Panel and API during the 12-hour maintenance period, and that DNS services were expected to resolve normally throughout the entire duration of the migration.

As the data center migration was underway, we experienced delays in transferring our database, which extended the duration of the work period and the length of time registration and management services were offline.

Alongside this issue, a bug in the new DNS provisioning system resulted in incomplete DNS records for up to 350,000 domains.

A handful of customers began reporting issues on Saturday, January 15, 2022 at 14:00 EST, which continued overnight. Enom engineers were actively resolving the reported DNS issues throughout Sunday, January 16, 2022, while a parallel effort was underway to determine the magnitude of the problem. All DNS resolution issues—including MX records—were resolved by January 17, 2022 at 13:45 EST.

At the same time, the following isolated issues arose:

Issue	Resolved
Inability to view, access, and search on domains under management via Enom Control Panel	Jan. 17, 2022 @ 13:00 EST
Credit card payment processing issues	Jan. 17, 2022 @ 21:00 EST
Enom Control Panel login issue	Jan. 18, 2022 @ 10:00 EST
Delays in system-generated email	Jan. 18, 2022 @ 12:00 EST

A cadence of incident updates was posted via Enom Status page, beginning Saturday, January 15, 2022, at 21:00 EST. We marked this outage as resolved on Tuesday, January 18, 2022, at 15:45 EST.

More information will be publicized in our incident report. We’ll distribute a summary of findings once complete.