The outage that is not yours

Sooner or later it is not your fault

At some point the thing that is broken is a certificate authority, a DNS provider, a cloud region, a payment gateway or an identity provider, and nothing in your repository can fix it. This has happened at industry scale several times: a configuration change at a single CDN taking a large share of the web offline for an hour, a DNS provider under attack making dozens of well-known services unreachable, a cloud region failing and taking down the products built on it, including, memorably, some of the status pages meant to report the failure.

Your job during those hours is not to fix somebody else’s system. It is to know quickly that it is theirs, to keep as much of your product working as you can, and to tell your users something true.

Establishing that it is them, in five minutes

A rough order that usually settles it:

1. Check from outside your own network. A phone on mobile data, or a free external checker. Half of “the site is down” reports are one office’s wifi or one ISP’s DNS resolver.
2. Check the dependency directly. curl its health endpoint or an unauthenticated URL from your server. Time it.
3. Check their status page and their social feed. Status pages lag, often by twenty minutes, because a human has to publish. The social feed is usually faster, and third-party outage trackers faster still.
4. Check your own dependency dashboard: the second screen from the dashboards lesson exists for precisely this moment.

Write the answer down with a timestamp as you go. In the postmortem, “we knew it was the provider at 14:07” is a much better sentence than “we think it was maybe an hour”.

What you can do, ranked by how much it actually helps

• Degrade the feature, keep the site. The failing-well lesson’s flag per dependency pays for itself here. Turning off search or recommendations is a ten-second action that keeps everything else alive.
• Queue the work. If the dependency is an email or webhook target, buffer and retry later. Users do not need those to be synchronous, and they rarely notice a twenty-minute delay.
• Serve stale. A cache configured with stale-if-error will keep answering from an old copy while the origin or an upstream is failing. This one line of configuration converts a total outage into slightly out-of-date content.
• Switch provider, if you already can. A second mail provider or model provider behind one interface is a cheap redundancy, because both are stateless HTTP APIs. This is realistic for mail, SMS, models and image processing.
• Wait, and communicate. For DNS, identity or your database’s region, there is often no alternative and pretending otherwise wastes the hour.

What is worth being redundant, and what is not

An honest ranking for a small project:

Worth it, because it is cheap: a second DNS provider (the protocol supports multiple authoritative servers), a second mail sender, a second model or API provider behind your own interface, backups stored at a different company from the database.

Not worth it for most: multi-region active-active, multi-cloud, a standby database on another provider. Each of these multiplies your cost and, more importantly, your complexity, and complexity causes outages. Teams that adopt multi-region for reliability often find their new failure modes outnumber the ones they removed. Single-region with good backups and an honest recovery time is a defensible choice, and saying so plainly is better engineering than an architecture you cannot operate.

The question to ask is not “could this fail” but “if this fails for four hours, what does it cost us, and is that more than the standing cost of the alternative”.

Say something, and say it somewhere they can reach

A status page hosted on the same infrastructure as your product is not a status page. Use a separate provider (several are free at small scale) or a static page on different hosting, or a social account. Whatever it is, it should be linked from your normal site so people know where to look before they need it.

The message during a third-party outage should say what is affected, what still works, that the cause is upstream, and when you will next update. It should not blame the vendor by way of excuse. Your users chose you, not them; the dependency was your decision.

Afterwards

Write it up like any other incident. The useful actions are almost never “switch provider” and almost always the smaller ones: add the flag that would have let you degrade, set stale-if-error, move the status page off the affected infrastructure, add the dependency to the dashboard you check first. Those are what shorten the next one.