Shipping ItLesson 46 of 74

Contents

Shipping It

Lesson 46 of 74. Sign in to keep your place.

Module 5 · Staying alive: the bad day and the day after

Writing down what happened

9 min read

A 42-minute outage, split the useful wayStart until somebodynoticed31Noticed until recovered11minutesThe bug was fixed in eleven minutes once anybody knew. The other thirty-one were ignorance, whichmeans the action that shortens the next incident is detection, not the bug.
The bug was fixed in eleven minutes once anybody knew. The other thirty-one were ignorance, which means the action that shortens the next incident is detection, not the bug.

Figure

The purpose is one thing

Not a record. Not accountability. A postmortem exists to make the next incident less likely or less long. Everything in its structure should serve that, and anything that does not is ceremony.

Blameless means something specific

It does not mean pretending nobody ran the command. It means the question is ”what made this easy to do and hard to notice?” rather than “who did it?”

The reasoning is practical rather than kind. In an organisation that punishes the person who caused an incident, people stop volunteering that they caused one. Detection time goes up, causes go unrecorded, and the same failure recurs with less information each time. Blamelessness is what keeps information flowing.

“Human error” is never a root cause. It is a place to start asking. Why was that command available? Why did nothing check? Why did the interface make the destructive option adjacent to the safe one?

The structure

Summary. Two sentences. What broke, for how long.

Impact, in user terms and in numbers. Not “some users saw errors”. Instead: 42 minutes, roughly 3,100 failed requests, 180 signups lost, checkout unavailable in Europe. Numbers make the priority argument for you afterwards; adjectives do not.

Timeline with timestamps. From your incident log. Include when it started, when it was detected, when the first action was taken, when it recovered.

What happened. The mechanism, plainly.

Contributing factors. Plural, always.

What went well. Real, not decorative: the alert fired correctly, the rollback worked, the runbook was accurate. Knowing which defences worked tells you which ones to keep investing in.

Actions. Owned and dated.

The most useful number is detection time

If the incident lasted 42 minutes and you learned about it at minute 31 from a customer, then the outage was 11 minutes of response and 31 minutes of ignorance. The thing to fix is detection, not the bug.

Teams reliably fix the bug and leave the detection gap in place, which is why the next unrelated bug also runs for half an hour. Split the duration into detect, decide and fix, every time. It tells you where your effort belongs.

Five whys is not enough

The technique of asking “why” five times produces a single causal chain, and real incidents are not chains. Three things were true at once: the check did not exist, the alert threshold was wrong, and the runbook was stale. A chain picks one and calls it the root cause.

Ask instead: ”what else had to be true for this to happen?” at each step. That produces a set of contributing factors, and it is a much better guide to what to change, because fixing any one of them may have been enough.

Three actions, owned, dated

A postmortem with fourteen unowned action items produces nothing at all, and worse, it teaches everyone that writing them is theatre. Pick the two or three that would most reduce recurrence or detection time. Give each a name and a date. Track them where your other work is tracked, not in the document.

One action that actually lands beats twelve that do not, by an infinite ratio.

Count them over a quarter

Individual postmortems fix individual incidents. Reading a quarter of them together shows the pattern, and the pattern is usually small: most teams find that three causes account for most of their incidents. That is a much better basis for deciding what to build than any individual document.

Even alone, write it

A docs/incidents/2027-01-14-login-outage.md costs twenty minutes. Six months later it is the only record of why the odd guard clause exists, why that timeout is 3 seconds and not 30, and why the deploy script refuses to run on a dirty tree. Your future self is a different person with none of your current context, and this is the cheapest way to send them a message.

Every lesson here is open. Your progress is recorded once you pass the Agents and Automation exam.

Check your understanding

An outage lasted 42 minutes. It began at 14:02, a customer emailed at 14:33, you rolled back at 14:41 and it recovered at 14:44. The postmortem’s single action item is a fix for the bug that caused it. What has been missed?

Pick the one you would defend. Your answer stays private.

Next lesson · 8 minBreaking it on purpose, on a Tuesday

Stuck on something?Ask Tri, your AI tutor

How to cite: Addaly (2026) ‘Writing down what happened’, in Shipping It. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/cloud-and-shipping/the-postmortem (Accessed: 13 October 2026).