Breaking it on purpose, on a Tuesday

Untested recovery is a belief, not a capability

You have backups. You have a rollback command. You have a runbook with the steps for a database failover. Every one of those is a claim about what will happen under conditions you have never created.

The backups lesson made this argument for restores. It generalises: every recovery mechanism you own is broken until somebody has used it while nothing was wrong. In practice, roughly half of them are broken the first time, and the reasons are dull: a credential that expired, a script that assumes a directory, a document that names a person who left, a command that needs a permission the on-call person does not have.

The fix is to rehearse, on a Tuesday morning, deliberately, with a rollback plan and a timer.

The five drills worth running, in order of value

1. Restore the database into a scratch environment. Time it end to end. Note the number, because that number is your real recovery time and it is usually two to five times what people guess. Quarterly.

2. Roll back a deploy. Not a hypothetical: actually deploy something harmless and roll it back. Measure the minutes. Monthly is easy because it costs nothing.

3. Kill an instance. Terminate one process or container while traffic flows. Do users notice? Does it come back on its own? This tests the supervisor, the health checks and the load balancer at once.

4. Take a dependency away. Block the mail provider or the model API with a firewall rule for five minutes. Does your site degrade or collapse? This is the drill that exposes missing timeouts, and it is the one that most often finds something.

5. Remove a person. Pick the person who knows the most and forbid them from touching the keyboard while somebody else follows the runbook. This is the cheapest test of documentation ever devised, and it is deeply uncomfortable, which is the point.

How to run one without causing the incident you were preparing for

• Announce it. Everybody who might see an alert knows the window. An unannounced drill teaches people to distrust their tools.
• Write the hypothesis first. “We believe killing one web instance will cause no visible errors.” A drill without a stated prediction is just poking things.
• Define the abort condition and who calls it, before you start.
• Start ridiculously small. One instance, in staging, in working hours. Production drills come after several successful staging ones.
• Timebox it. Thirty minutes, including the write-up.
• Write down what broke, and fix it that week. A drill that finds three problems and fixes none is a way of feeling prepared.

Chaos engineering, sized honestly

The industrial version of this (automatically terminating instances in production, injecting latency, running continuously) came from organisations with hundreds of services and full-time reliability teams. The tooling exists and is partly free, but for a project with one or two services it is a distraction.

What scales down perfectly is the method: form a hypothesis about steady-state behaviour, introduce one real fault, observe whether the hypothesis held, fix the gap. You need a terminal and a calendar reminder, not a platform.

The game day, once a quarter

A game day is a scripted incident. Somebody writes a scenario (“the primary database is unreachable, it is 09:00 on a Monday”) and everybody else works through it using only the real tools and documents. In its cheapest form, nothing is actually broken: people open the runbook, say what they would type, and discover that the runbook does not say where the credentials live.

For a solo maintainer this works as an hour with a notebook. Read your own runbook as though you had never seen it, follow it literally, and write down every point at which you had to use knowledge that is not written down. That list is your documentation backlog, and it is more accurate than any amount of reflecting on what might be missing.

The number to keep

After each drill, record two figures: how long the recovery took, and how many surprises it produced. Both should fall over a year. If your restore time has been “about an hour, probably” for three years, you do not have a recovery time: you have a guess that has never been contradicted.