The first thirty minutes of an outage

The order matters more than the skill

Under pressure, people debug first and communicate never. Both are backwards. Here is a sequence that works, and it is worth reading before you need it, because you will not be reading anything at the time.

1. Confirm it, and scope it: two minutes

“The site is down” from one person is sometimes their office wifi or their DNS resolver. Check from outside: your phone on mobile data, a different network, your external uptime monitor, a public checking service.

Then scope it. All users or some? All pages or one? All regions? Signed-in only? The scope is the strongest clue you will get in the first ten minutes: “only signed-in users, only in Europe” points somewhere very specific, and it costs a minute to establish.

2. Say something: within five minutes

One line, early, buys you an hour of patience:

We are aware of errors affecting logins and are investigating. Next update in 20 minutes.

Three properties make it work: what is affected, that a human is on it, and when you will speak again. Then speak again at that time even if the news is “still working on it”. A missed promised update costs more trust than the outage itself.

Host your status page somewhere else. A status page on the infrastructure that is down is a joke you make exactly once. A static page on GitHub Pages or Netlify, or a pinned post on whatever social account you have, costs nothing and works when nothing else does.

3. Stop the bleeding before you understand it

This is the step people skip and it is the most important one.

Roll back. Turn off the feature flag. Disable the expensive endpoint. Take the broken feature offline and leave the rest of the site working. You do not need to know the cause to do any of these. And none of them destroy the evidence, because the logs, the metrics and the bad artifact all still exist.

The most common way a fifteen-minute incident becomes a two-hour one is somebody choosing to understand it first. Curiosity is the right instinct at the wrong moment. End it, then be curious.

4. Ask what changed

Most incidents are a change. Check, in this order:

• Deploys and migrations in the last few hours.
• Config and feature-flag changes, often made by someone who does not think of it as a deploy.
• Your vendors’ status pages.
• Anything that expired: a certificate, a card, a key.
• A traffic change: a spike, a crawler, a marketing email that just went out.

If nothing changed on your side, it changed on someone else’s. Check the vendor status pages early, because five minutes of reading beats forty of debugging code that is fine.

5. One coordinator, one channel, timestamps

Even alone, keep a running log as you go:

14:02 alerted, 5xx on /login ~40%
14:05 posted status update
14:08 rolled back to build 8f2a1c
14:11 error rate falling
14:19 clear. cause not yet known

Two minutes of typing during the event, and you have the postmortem timeline for free. You will not remember this accurately at 4am, and reconstructing it afterwards from logs takes an hour.

With more than one person, name a coordinator whose job is deciding and communicating rather than typing commands. Two people debugging in parallel without coordination will undo each other’s changes.

6. Change one thing at a time

Under pressure the urge is to try five fixes at once. Resist it. If five changes fix it, you do not know which one worked, so you cannot write it down, cannot prevent it, and are carrying four unexplained changes into next week, one of which may break something else on Thursday.

What not to do

• Do not restart everything reflexively. It sometimes helps and it destroys the state you needed to diagnose, which guarantees the same incident next month.
• Do not scale up to hide a bug. Sometimes the right emergency move, but write down that you did it, or the leak stays hidden behind larger machines and the bill.
• Do not silence the alert. Acknowledge it. Silencing is how the second, worse incident goes unnoticed.
• Do not blame anyone during the incident. It is the least useful thing available and it delays the moment someone says “I think I did this”, which is often the fastest route to the fix.

Prepare one thing

A runbook, written on a calm afternoon: the rollback command, the vendor status URLs, where the logs are, how to reach the database, who else can help, and the login for the status page. One page.

The value is not the information: you know most of it. The value is that at 3am you are not reconstructing it from memory while people wait.