{"course":{"slug":"cloud-and-shipping","title":"Shipping It","url":"https://addaly.com/learn/cloud-and-shipping","level":"intermediate","track":"build"},"lesson":{"slug":"the-first-thirty-minutes","title":"The first thirty minutes of an outage","url":"https://addaly.com/learn/cloud-and-shipping/the-first-thirty-minutes","position":45,"of":74,"minutes":10,"answer":"Stop the bleeding before you understand the cause, because diagnosis works fine from logs and a rolled-back artifact while the fire does not need to keep burning.","keyIdea":"Stop the bleeding before you understand the cause, because diagnosis works fine from logs and a rolled-back artifact while the fire does not need to keep burning."},"body":{"format":"markdown","text":"## The order matters more than the skill\n\nUnder pressure, people debug first and communicate never. Both are backwards. Here is a sequence that works, and it is worth reading before you need it, because you will not be reading anything at the time.\n\n## 1. Confirm it, and scope it: two minutes\n\n\"The site is down\" from one person is sometimes their office wifi or their DNS resolver. Check from outside: your phone on mobile data, a different network, your external uptime monitor, a public checking service.\n\nThen scope it. All users or some? All pages or one? All regions? Signed-in only? The scope is the strongest clue you will get in the first ten minutes: \"only signed-in users, only in Europe\" points somewhere very specific, and it costs a minute to establish.\n\n## 2. Say something: within five minutes\n\nOne line, early, buys you an hour of patience:\n\n> We are aware of errors affecting logins and are investigating. Next update in 20 minutes.\n\nThree properties make it work: what is affected, that a human is on it, and **when you will speak again**. Then speak again at that time even if the news is \"still working on it\". A missed promised update costs more trust than the outage itself.\n\nHost your status page **somewhere else**. A status page on the infrastructure that is down is a joke you make exactly once. A static page on GitHub Pages or Netlify, or a pinned post on whatever social account you have, costs nothing and works when nothing else does.\n\n## 3. Stop the bleeding before you understand it\n\nThis is the step people skip and it is the most important one.\n\nRoll back. Turn off the feature flag. Disable the expensive endpoint. Take the broken feature offline and leave the rest of the site working. You do not need to know the cause to do any of these. And none of them destroy the evidence, because the logs, the metrics and the bad artifact all still exist.\n\nThe most common way a fifteen-minute incident becomes a two-hour one is somebody choosing to understand it first. Curiosity is the right instinct at the wrong moment. End it, then be curious.\n\n## 4. Ask what changed\n\nMost incidents are a change. Check, in this order:\n\n- Deploys and migrations in the last few hours.\n- Config and feature-flag changes, often made by someone who does not think of it as a deploy.\n- Your vendors' status pages.\n- Anything that expired: a certificate, a card, a key.\n- A traffic change: a spike, a crawler, a marketing email that just went out.\n\nIf nothing changed on your side, it changed on someone else's. Check the vendor status pages early, because five minutes of reading beats forty of debugging code that is fine.\n\n## 5. One coordinator, one channel, timestamps\n\nEven alone, keep a running log as you go:\n\n```\n14:02 alerted, 5xx on /login ~40%\n14:05 posted status update\n14:08 rolled back to build 8f2a1c\n14:11 error rate falling\n14:19 clear. cause not yet known\n```\n\nTwo minutes of typing during the event, and you have the postmortem timeline for free. You will not remember this accurately at 4am, and reconstructing it afterwards from logs takes an hour.\n\nWith more than one person, name a coordinator whose job is deciding and communicating rather than typing commands. Two people debugging in parallel without coordination will undo each other's changes.\n\n## 6. Change one thing at a time\n\nUnder pressure the urge is to try five fixes at once. Resist it. If five changes fix it, you do not know which one worked, so you cannot write it down, cannot prevent it, and are carrying four unexplained changes into next week, one of which may break something else on Thursday.\n\n## What not to do\n\n- **Do not restart everything reflexively.** It sometimes helps and it destroys the state you needed to diagnose, which guarantees the same incident next month.\n- **Do not scale up to hide a bug.** Sometimes the right emergency move, but write down that you did it, or the leak stays hidden behind larger machines and the bill.\n- **Do not silence the alert.** Acknowledge it. Silencing is how the second, worse incident goes unnoticed.\n- **Do not blame anyone during the incident.** It is the least useful thing available and it delays the moment someone says \"I think I did this\", which is often the fastest route to the fix.\n\n## Prepare one thing\n\nA runbook, written on a calm afternoon: the rollback command, the vendor status URLs, where the logs are, how to reach the database, who else can help, and the login for the status page. One page.\n\nThe value is not the information: you know most of it. The value is that at 3am you are not reconstructing it from memory while people wait."},"check":{"question":"Twenty minutes after a deploy, error rates jump to 30% and you have no idea why. Your rollback takes about three minutes. What should you do first?","options":[{"text":"Check the logs to identify the failing code path so the rollback is informed","correct":false,"tempting":"Sounds diligent and is the most common way a short incident becomes a long one; the logs will still be there after the site is working."},{"text":"Scale up the service, in case the new version is resource-hungry","correct":false,"tempting":"Guesses at a cause, spends money, and leaves the broken version serving users while you wait to see whether the guess was right."},{"text":"Roll back immediately, confirm recovery, then diagnose from the logs and the failed artifact","correct":true},{"text":"Post a status update first, then investigate before deciding whether to roll back","correct":false,"tempting":"Communication matters and belongs in the first five minutes, but it takes seconds and does not compete with the rollback; delaying the fix in order to investigate is the error."}]},"prev":{"slug":"when-your-provider-goes-down","title":"The outage that is not yours","url":"https://addaly.com/learn/cloud-and-shipping/when-your-provider-goes-down"},"next":{"slug":"the-postmortem","title":"Writing down what happened","url":"https://addaly.com/learn/cloud-and-shipping/the-postmortem"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The first thirty minutes of an outage', in Shipping It. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/cloud-and-shipping/the-first-thirty-minutes (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The first thirty minutes of an outage","courseTitle":"Shipping It","module":{"index":5,"slug":"staying-alive","title":"Staying alive: the bad day and the day after"},"answer":null,"keyIdea":"Stop the bleeding before you understand the cause, because diagnosis works fine from logs and a rolled-back artifact while the fire does not need to keep burning.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-order-matters-more-than-the-skill","runs":[{"text":"The order matters more than the skill"}]},{"t":"p","runs":[{"text":"Under pressure, people debug first and communicate never. Both are backwards. Here is a sequence that works, and it is worth reading before you need it, because you will not be reading anything at the time."}]},{"t":"h2","id":"1-confirm-it-and-scope-it-two-minutes","runs":[{"text":"1. Confirm it, and scope it: two minutes"}]},{"t":"p","runs":[{"text":"“The site is down” from one person is sometimes their office wifi or their DNS resolver. Check from outside: your phone on mobile data, a different network, your external uptime monitor, a public checking service."}]},{"t":"p","runs":[{"text":"Then scope it. All users or some? All pages or one? All regions? Signed-in only? The scope is the strongest clue you will get in the first ten minutes: “only signed-in users, only in Europe” points somewhere very specific, and it costs a minute to establish."}]},{"t":"h2","id":"2-say-something-within-five-minutes","runs":[{"text":"2. Say something: within five minutes"}]},{"t":"p","runs":[{"text":"One line, early, buys you an hour of patience:"}]},{"t":"quote","runs":[{"text":"We are aware of errors affecting logins and are investigating. Next update in 20 minutes."}]},{"t":"p","runs":[{"text":"Three properties make it work: what is affected, that a human is on it, and "},{"text":"when you will speak again","b":true},{"text":". Then speak again at that time even if the news is “still working on it”. A missed promised update costs more trust than the outage itself."}]},{"t":"p","runs":[{"text":"Host your status page "},{"text":"somewhere else","b":true},{"text":". A status page on the infrastructure that is down is a joke you make exactly once. A static page on GitHub Pages or Netlify, or a pinned post on whatever social account you have, costs nothing and works when nothing else does."}]},{"t":"h2","id":"3-stop-the-bleeding-before-you-understand-it","runs":[{"text":"3. Stop the bleeding before you understand it"}]},{"t":"p","runs":[{"text":"This is the step people skip and it is the most important one."}]},{"t":"p","runs":[{"text":"Roll back. Turn off the feature flag. Disable the expensive endpoint. Take the broken feature offline and leave the rest of the site working. You do not need to know the cause to do any of these. And none of them destroy the evidence, because the logs, the metrics and the bad artifact all still exist."}]},{"t":"p","runs":[{"text":"The most common way a fifteen-minute incident becomes a two-hour one is somebody choosing to understand it first. Curiosity is the right instinct at the wrong moment. End it, then be curious."}]},{"t":"h2","id":"4-ask-what-changed","runs":[{"text":"4. Ask what changed"}]},{"t":"p","runs":[{"text":"Most incidents are a change. Check, in this order:"}]},{"t":"ul","items":[[{"text":"Deploys and migrations in the last few hours."}],[{"text":"Config and feature-flag changes, often made by someone who does not think of it as a deploy."}],[{"text":"Your vendors’ status pages."}],[{"text":"Anything that expired: a certificate, a card, a key."}],[{"text":"A traffic change: a spike, a crawler, a marketing email that just went out."}]]},{"t":"p","runs":[{"text":"If nothing changed on your side, it changed on someone else’s. Check the vendor status pages early, because five minutes of reading beats forty of debugging code that is fine."}]},{"t":"h2","id":"5-one-coordinator-one-channel-timestamps","runs":[{"text":"5. One coordinator, one channel, timestamps"}]},{"t":"p","runs":[{"text":"Even alone, keep a running log as you go:"}]},{"t":"code","lang":"","label":"Text","text":"14:02 alerted, 5xx on /login ~40%\n14:05 posted status update\n14:08 rolled back to build 8f2a1c\n14:11 error rate falling\n14:19 clear. cause not yet known"},{"t":"p","runs":[{"text":"Two minutes of typing during the event, and you have the postmortem timeline for free. You will not remember this accurately at 4am, and reconstructing it afterwards from logs takes an hour."}]},{"t":"p","runs":[{"text":"With more than one person, name a coordinator whose job is deciding and communicating rather than typing commands. Two people debugging in parallel without coordination will undo each other’s changes."}]},{"t":"h2","id":"6-change-one-thing-at-a-time","runs":[{"text":"6. Change one thing at a time"}]},{"t":"p","runs":[{"text":"Under pressure the urge is to try five fixes at once. Resist it. If five changes fix it, you do not know which one worked, so you cannot write it down, cannot prevent it, and are carrying four unexplained changes into next week, one of which may break something else on Thursday."}]},{"t":"h2","id":"what-not-to-do","runs":[{"text":"What not to do"}]},{"t":"ul","items":[[{"text":"Do not restart everything reflexively.","b":true},{"text":" It sometimes helps and it destroys the state you needed to diagnose, which guarantees the same incident next month."}],[{"text":"Do not scale up to hide a bug.","b":true},{"text":" Sometimes the right emergency move, but write down that you did it, or the leak stays hidden behind larger machines and the bill."}],[{"text":"Do not silence the alert.","b":true},{"text":" Acknowledge it. Silencing is how the second, worse incident goes unnoticed."}],[{"text":"Do not blame anyone during the incident.","b":true},{"text":" It is the least useful thing available and it delays the moment someone says “I think I did this”, which is often the fastest route to the fix."}]]},{"t":"h2","id":"prepare-one-thing","runs":[{"text":"Prepare one thing"}]},{"t":"p","runs":[{"text":"A runbook, written on a calm afternoon: the rollback command, the vendor status URLs, where the logs are, how to reach the database, who else can help, and the login for the status page. One page."}]},{"t":"p","runs":[{"text":"The value is not the information: you know most of it. The value is that at 3am you are not reconstructing it from memory while people wait."}]}],"check":{"question":[{"text":"Twenty minutes after a deploy, error rates jump to 30% and you have no idea why. Your rollback takes about three minutes. What should you do first?"}],"options":[{"text":[{"text":"Check the logs to identify the failing code path so the rollback is informed"}],"correct":false,"tempting":[{"text":"Sounds diligent and is the most common way a short incident becomes a long one; the logs will still be there after the site is working."}]},{"text":[{"text":"Scale up the service, in case the new version is resource-hungry"}],"correct":false,"tempting":[{"text":"Guesses at a cause, spends money, and leaves the broken version serving users while you wait to see whether the guess was right."}]},{"text":[{"text":"Roll back immediately, confirm recovery, then diagnose from the logs and the failed artifact"}],"correct":true,"tempting":null},{"text":[{"text":"Post a status update first, then investigate before deciding whether to roll back"}],"correct":false,"tempting":[{"text":"Communication matters and belongs in the first five minutes, but it takes seconds and does not compete with the rollback; delaying the fix in order to investigate is the error."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"when-your-provider-goes-down","title":"The outage that is not yours","minutes":8},"next":{"slug":"the-postmortem","title":"Writing down what happened","minutes":9}}}