{"course":{"slug":"cloud-and-shipping","title":"Shipping It","url":"https://addaly.com/learn/cloud-and-shipping","level":"intermediate","track":"build"},"lesson":{"slug":"practising-failure","title":"Breaking it on purpose, on a Tuesday","url":"https://addaly.com/learn/cloud-and-shipping/practising-failure","position":47,"of":74,"minutes":8,"answer":"Every recovery mechanism is a belief until somebody has used it deliberately while nothing was wrong, so rehearse restores, rollbacks.","keyIdea":"Every recovery mechanism is a belief until somebody has used it deliberately while nothing was wrong, so rehearse restores, rollbacks, instance kills and dependency failures on a calm Tuesday and record how long each really took."},"body":{"format":"markdown","text":"## Untested recovery is a belief, not a capability\n\nYou have backups. You have a rollback command. You have a runbook with the steps for a database failover. Every one of those is a claim about what will happen under conditions you have never created.\n\nThe backups lesson made this argument for restores. It generalises: every recovery mechanism you own is broken until somebody has used it while nothing was wrong. In practice, roughly half of them are broken the first time, and the reasons are dull: a credential that expired, a script that assumes a directory, a document that names a person who left, a command that needs a permission the on-call person does not have.\n\nThe fix is to rehearse, on a Tuesday morning, deliberately, with a rollback plan and a timer.\n\n## The five drills worth running, in order of value\n\n**1. Restore the database into a scratch environment.** Time it end to end. Note the number, because that number is your real recovery time and it is usually two to five times what people guess. Quarterly.\n\n**2. Roll back a deploy.** Not a hypothetical: actually deploy something harmless and roll it back. Measure the minutes. Monthly is easy because it costs nothing.\n\n**3. Kill an instance.** Terminate one process or container while traffic flows. Do users notice? Does it come back on its own? This tests the supervisor, the health checks and the load balancer at once.\n\n**4. Take a dependency away.** Block the mail provider or the model API with a firewall rule for five minutes. Does your site degrade or collapse? This is the drill that exposes missing timeouts, and it is the one that most often finds something.\n\n**5. Remove a person.** Pick the person who knows the most and forbid them from touching the keyboard while somebody else follows the runbook. This is the cheapest test of documentation ever devised, and it is deeply uncomfortable, which is the point.\n\n## How to run one without causing the incident you were preparing for\n\n- **Announce it.** Everybody who might see an alert knows the window. An unannounced drill teaches people to distrust their tools.\n- **Write the hypothesis first.** \"We believe killing one web instance will cause no visible errors.\" A drill without a stated prediction is just poking things.\n- **Define the abort condition and who calls it**, before you start.\n- **Start ridiculously small.** One instance, in staging, in working hours. Production drills come after several successful staging ones.\n- **Timebox it.** Thirty minutes, including the write-up.\n- **Write down what broke, and fix it that week.** A drill that finds three problems and fixes none is a way of feeling prepared.\n\n## Chaos engineering, sized honestly\n\nThe industrial version of this (automatically terminating instances in production, injecting latency, running continuously) came from organisations with hundreds of services and full-time reliability teams. The tooling exists and is partly free, but for a project with one or two services it is a distraction.\n\nWhat scales down perfectly is the *method*: form a hypothesis about steady-state behaviour, introduce one real fault, observe whether the hypothesis held, fix the gap. You need a terminal and a calendar reminder, not a platform.\n\n## The game day, once a quarter\n\nA game day is a scripted incident. Somebody writes a scenario (\"the primary database is unreachable, it is 09:00 on a Monday\") and everybody else works through it using only the real tools and documents. In its cheapest form, nothing is actually broken: people open the runbook, say what they would type, and discover that the runbook does not say where the credentials live.\n\nFor a solo maintainer this works as an hour with a notebook. Read your own runbook as though you had never seen it, follow it literally, and write down every point at which you had to use knowledge that is not written down. That list is your documentation backlog, and it is more accurate than any amount of reflecting on what might be missing.\n\n## The number to keep\n\nAfter each drill, record two figures: how long the recovery took, and how many surprises it produced. Both should fall over a year. If your restore time has been \"about an hour, probably\" for three years, you do not have a recovery time: you have a guess that has never been contradicted."},"check":{"question":"A team runs a drill: they block their mail provider for five minutes. The site keeps working, but every page takes 8 seconds to load. What has the drill demonstrated?","options":[{"text":"The mail provider is a hard dependency and the site should fail closed instead","correct":false,"tempting":"Failing closed would make things worse; the site correctly continued to serve, and the fault is in how long it waits, not in whether it proceeds."},{"text":"Blocking at the firewall is an unrealistic fault, since real outages return errors rather than hanging","correct":false,"tempting":"Real outages very often do hang (packets dropped silently is the common case) which is exactly why this drill is representative."},{"text":"A call to the mail provider sits somewhere in the page's critical path with a long or absent timeout","correct":true},{"text":"The retry policy is too aggressive, adding delay on every request","correct":false,"tempting":"Retries would multiply the delay, but the primary evidence is a uniform hang, which points at a single long wait before any retry logic matters."}]},"prev":{"slug":"the-postmortem","title":"Writing down what happened","url":"https://addaly.com/learn/cloud-and-shipping/the-postmortem"},"next":{"slug":"handing-it-over","title":"Writing it down for whoever is next","url":"https://addaly.com/learn/cloud-and-shipping/handing-it-over"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Breaking it on purpose, on a Tuesday', in Shipping It. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/cloud-and-shipping/practising-failure (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Breaking it on purpose, on a Tuesday","courseTitle":"Shipping It","module":{"index":5,"slug":"staying-alive","title":"Staying alive: the bad day and the day after"},"answer":null,"keyIdea":"Every recovery mechanism is a belief until somebody has used it deliberately while nothing was wrong, so rehearse restores, rollbacks, instance kills and dependency failures on a calm Tuesday and record how long each really took.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"untested-recovery-is-a-belief-not-a-capability","runs":[{"text":"Untested recovery is a belief, not a capability"}]},{"t":"p","runs":[{"text":"You have backups. You have a rollback command. You have a runbook with the steps for a database failover. Every one of those is a claim about what will happen under conditions you have never created."}]},{"t":"p","runs":[{"text":"The backups lesson made this argument for restores. It generalises: every recovery mechanism you own is broken until somebody has used it while nothing was wrong. In practice, roughly half of them are broken the first time, and the reasons are dull: a credential that expired, a script that assumes a directory, a document that names a person who left, a command that needs a permission the on-call person does not have."}]},{"t":"p","runs":[{"text":"The fix is to rehearse, on a Tuesday morning, deliberately, with a rollback plan and a timer."}]},{"t":"h2","id":"the-five-drills-worth-running-in-order-of-value","runs":[{"text":"The five drills worth running, in order of value"}]},{"t":"p","runs":[{"text":"1. Restore the database into a scratch environment.","b":true},{"text":" Time it end to end. Note the number, because that number is your real recovery time and it is usually two to five times what people guess. Quarterly."}]},{"t":"p","runs":[{"text":"2. Roll back a deploy.","b":true},{"text":" Not a hypothetical: actually deploy something harmless and roll it back. Measure the minutes. Monthly is easy because it costs nothing."}]},{"t":"p","runs":[{"text":"3. Kill an instance.","b":true},{"text":" Terminate one process or container while traffic flows. Do users notice? Does it come back on its own? This tests the supervisor, the health checks and the load balancer at once."}]},{"t":"p","runs":[{"text":"4. Take a dependency away.","b":true},{"text":" Block the mail provider or the model API with a firewall rule for five minutes. Does your site degrade or collapse? This is the drill that exposes missing timeouts, and it is the one that most often finds something."}]},{"t":"p","runs":[{"text":"5. Remove a person.","b":true},{"text":" Pick the person who knows the most and forbid them from touching the keyboard while somebody else follows the runbook. This is the cheapest test of documentation ever devised, and it is deeply uncomfortable, which is the point."}]},{"t":"h2","id":"how-to-run-one-without-causing-the-incident-you-were-prepari","runs":[{"text":"How to run one without causing the incident you were preparing for"}]},{"t":"ul","items":[[{"text":"Announce it.","b":true},{"text":" Everybody who might see an alert knows the window. An unannounced drill teaches people to distrust their tools."}],[{"text":"Write the hypothesis first.","b":true},{"text":" “We believe killing one web instance will cause no visible errors.” A drill without a stated prediction is just poking things."}],[{"text":"Define the abort condition and who calls it","b":true},{"text":", before you start."}],[{"text":"Start ridiculously small.","b":true},{"text":" One instance, in staging, in working hours. Production drills come after several successful staging ones."}],[{"text":"Timebox it.","b":true},{"text":" Thirty minutes, including the write-up."}],[{"text":"Write down what broke, and fix it that week.","b":true},{"text":" A drill that finds three problems and fixes none is a way of feeling prepared."}]]},{"t":"h2","id":"chaos-engineering-sized-honestly","runs":[{"text":"Chaos engineering, sized honestly"}]},{"t":"p","runs":[{"text":"The industrial version of this (automatically terminating instances in production, injecting latency, running continuously) came from organisations with hundreds of services and full-time reliability teams. The tooling exists and is partly free, but for a project with one or two services it is a distraction."}]},{"t":"p","runs":[{"text":"What scales down perfectly is the "},{"text":"method","i":true},{"text":": form a hypothesis about steady-state behaviour, introduce one real fault, observe whether the hypothesis held, fix the gap. You need a terminal and a calendar reminder, not a platform."}]},{"t":"h2","id":"the-game-day-once-a-quarter","runs":[{"text":"The game day, once a quarter"}]},{"t":"p","runs":[{"text":"A game day is a scripted incident. Somebody writes a scenario (“the primary database is unreachable, it is 09:00 on a Monday”) and everybody else works through it using only the real tools and documents. In its cheapest form, nothing is actually broken: people open the runbook, say what they would type, and discover that the runbook does not say where the credentials live."}]},{"t":"p","runs":[{"text":"For a solo maintainer this works as an hour with a notebook. Read your own runbook as though you had never seen it, follow it literally, and write down every point at which you had to use knowledge that is not written down. That list is your documentation backlog, and it is more accurate than any amount of reflecting on what might be missing."}]},{"t":"h2","id":"the-number-to-keep","runs":[{"text":"The number to keep"}]},{"t":"p","runs":[{"text":"After each drill, record two figures: how long the recovery took, and how many surprises it produced. Both should fall over a year. If your restore time has been “about an hour, probably” for three years, you do not have a recovery time: you have a guess that has never been contradicted."}]}],"check":{"question":[{"text":"A team runs a drill: they block their mail provider for five minutes. The site keeps working, but every page takes 8 seconds to load. What has the drill demonstrated?"}],"options":[{"text":[{"text":"The mail provider is a hard dependency and the site should fail closed instead"}],"correct":false,"tempting":[{"text":"Failing closed would make things worse; the site correctly continued to serve, and the fault is in how long it waits, not in whether it proceeds."}]},{"text":[{"text":"Blocking at the firewall is an unrealistic fault, since real outages return errors rather than hanging"}],"correct":false,"tempting":[{"text":"Real outages very often do hang (packets dropped silently is the common case) which is exactly why this drill is representative."}]},{"text":[{"text":"A call to the mail provider sits somewhere in the page’s critical path with a long or absent timeout"}],"correct":true,"tempting":null},{"text":[{"text":"The retry policy is too aggressive, adding delay on every request"}],"correct":false,"tempting":[{"text":"Retries would multiply the delay, but the primary evidence is a uniform hang, which points at a single long wait before any retry logic matters."}]}],"order":[1,0,2,3],"why":null},"prev":{"slug":"the-postmortem","title":"Writing down what happened","minutes":9},"next":{"slug":"handing-it-over","title":"Writing it down for whoever is next","minutes":9}}}