{"course":{"slug":"how-software-changed","title":"How Software Development Changed","url":"https://addaly.com/learn/how-software-changed","level":"intermediate","track":"work"},"lesson":{"slug":"incidents-and-on-call","title":"Operating a system nobody wrote by hand","url":"https://addaly.com/learn/how-software-changed/incidents-and-on-call","position":58,"of":87,"minutes":9,"answer":"When nobody remembers writing the code, the first question at 3am becomes what changed in the last day, so build the change timeline and stamp every log line with the deploy version.","keyIdea":"When nobody remembers writing the code, the first question at 3am becomes what changed in the last day, so build the change timeline and stamp every log line with the deploy version, and convert every incident into one permanent test in the same week."},"body":{"format":"markdown","text":"## At 3am, the useful question changed\n\nThe old first question was \"who wrote this?\", because that person could say \"check the cache invalidation, it is always the cache invalidation\", and forty minutes became four.\n\nThat question now often has no useful answer. The replacement is mechanical and works regardless of provenance: **what changed in the last twenty-four hours?**\n\nDeployments, feature flags, configuration values, dependency updates, database migrations, infrastructure changes, and things that changed on somebody else's side. A single timeline showing all of these, in one place, is the highest-value observability investment a team can make, and most teams have the data spread across five systems and correlate it by hand under pressure.\n\nBuild the timeline before you need it. During an incident it is worth an hour of anybody's time; at the moment you need it you have four minutes.\n\n## What to log so the question is answerable\n\nThe general rule: log the values at the boundaries, in a structure a machine can filter.\n\n```python\nlogger.info(\"charge.attempt\", extra={\n    \"request_id\": ctx.request_id,\n    \"account_id\": account.id,\n    \"amount_minor\": amount,\n    \"currency\": currency,\n    \"idempotency_key\": key,\n    \"provider\": \"razorpay\",\n    \"deploy_sha\": settings.GIT_SHA,\n})\n```\n\n`deploy_sha` on every log line is the detail most often missing and the one that most often resolves an incident, because it lets you filter to the version and answer the question above directly.\n\nThe three that matter most: a request identifier that travels across services, the inputs and outputs at every external boundary, and the version. Everything else is a refinement.\n\n## Runbooks, written for the person who does not know\n\nA runbook assuming familiarity is useless at 3am to somebody who does not have it, which, increasingly, is whoever is on call.\n\nOne page per alert:\n\n```markdown\n# Alert: checkout_error_rate > 2%\n\nWhat it means: more than 2% of POST /checkout returned 5xx over 5 minutes.\nWho it affects: customers cannot complete purchases. Revenue-affecting.\n\nFirst: check https://dash/deploys — anything in the last hour?\n        If yes, roll back first and diagnose afterwards.\n\nThen:  check the provider status page (link).\n       Run: `kubectl logs -l app=checkout --since=10m | grep -c 'provider timeout'`\n       If provider timeouts dominate, set flag `checkout.fallback_provider=true`.\n\nEscalate to: #payments-oncall. Ravi knows the reconciliation path.\nDo NOT: re-run the settlement job. It is not idempotent. See ADR 0031.\n```\n\nThe *do not* section is the one people skip and the one that prevents the second, worse incident.\n\n## Roll back first\n\nWith more change flowing and less of it deeply understood, the balance shifted further towards reverting before diagnosing. Diagnosis takes as long as it takes; a rollback takes minutes and stops the bleeding.\n\nThe practices that make this available: deploy small, deploy often, keep the previous version ready, make migrations backwards compatible for at least one release, and keep flags for anything risky. These are the same practices that make everything else in this course work, which is not a coincidence: reversibility is the property that makes speed survivable at every level.\n\n## The review, and the one output that matters\n\nBlameless, because a review that assigns fault produces careful reporting rather than accurate reporting, and you need the accurate kind.\n\nOne mechanical change worth making: the standard prompt has always been \"what did the person do and why was it reasonable at the time?\" Extend it to \"what did the system present to the person, and why was that presentation reasonable to act on?\" A generated change that looked correct, passed review, and was wrong is a systems failure with several contributing causes, and hunting for the individual who should have caught it finds nothing useful.\n\nAnd the output that matters is not the document. It is **a permanent test.**\n\nEvery incident produces one, in the same week, before anybody has moved on. A property, a database constraint, a monitor, an assertion, an item on the hostile-inputs corpus. The suite that catches real problems is built almost entirely out of things that actually happened, and a team that does this consistently has, after two years, a test suite worth more than anything anyone could have designed in advance.\n\nAn incident review with no test attached is a document. An incident review with a test attached is verification capacity, which is the constrained resource this entire course is about."},"check":{"question":"A team's incident reviews are thorough, blameless and well written, yet the same class of failure recurs three times in a year. What is most likely missing?","options":[{"text":"Follow-up action items with owners and deadlines, since well-written reviews often produce recommendations nobody executes.","correct":false,"tempting":"Close, and better than nothing, but an action item is a promise while a test is an enforcement that cannot be quietly dropped."},{"text":"Each review produced a document but no permanent automated check.","correct":true},{"text":"The reviews are too blameless, so no individual felt responsible for preventing a recurrence.","correct":false,"tempting":"Reintroduces blame as a control, which produces more careful reporting rather than fewer failures."},{"text":"The root cause analysis stopped too early, identifying a proximate trigger rather than the underlying systemic condition.","correct":false,"tempting":"A real weakness in shallow reviews, but the description says they are thorough: the gap is between understanding and enforcement."}]},"prev":{"slug":"onboarding-without-an-author","title":"Onboarding into code nobody wrote","url":"https://addaly.com/learn/how-software-changed/onboarding-without-an-author"},"next":{"slug":"a-tool-policy-people-follow","title":"A policy people will actually follow","url":"https://addaly.com/learn/how-software-changed/a-tool-policy-people-follow"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Operating a system nobody wrote by hand', in How Software Development Changed. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-software-changed/incidents-and-on-call (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Operating a system nobody wrote by hand","courseTitle":"How Software Development Changed","module":{"index":6,"slug":"teams-and-delivery","title":"Teams, process and delivery"},"answer":null,"keyIdea":"When nobody remembers writing the code, the first question at 3am becomes what changed in the last day, so build the change timeline and stamp every log line with the deploy version, and convert every incident into one permanent test in the same week.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"at-3am-the-useful-question-changed","runs":[{"text":"At 3am, the useful question changed"}]},{"t":"p","runs":[{"text":"The old first question was “who wrote this?”, because that person could say “check the cache invalidation, it is always the cache invalidation”, and forty minutes became four."}]},{"t":"p","runs":[{"text":"That question now often has no useful answer. The replacement is mechanical and works regardless of provenance: "},{"text":"what changed in the last twenty-four hours?","b":true}]},{"t":"p","runs":[{"text":"Deployments, feature flags, configuration values, dependency updates, database migrations, infrastructure changes, and things that changed on somebody else’s side. A single timeline showing all of these, in one place, is the highest-value observability investment a team can make, and most teams have the data spread across five systems and correlate it by hand under pressure."}]},{"t":"p","runs":[{"text":"Build the timeline before you need it. During an incident it is worth an hour of anybody’s time; at the moment you need it you have four minutes."}]},{"t":"h2","id":"what-to-log-so-the-question-is-answerable","runs":[{"text":"What to log so the question is answerable"}]},{"t":"p","runs":[{"text":"The general rule: log the values at the boundaries, in a structure a machine can filter."}]},{"t":"code","lang":"python","label":"Python","text":"logger.info(\"charge.attempt\", extra={\n    \"request_id\": ctx.request_id,\n    \"account_id\": account.id,\n    \"amount_minor\": amount,\n    \"currency\": currency,\n    \"idempotency_key\": key,\n    \"provider\": \"razorpay\",\n    \"deploy_sha\": settings.GIT_SHA,\n})"},{"t":"p","runs":[{"text":"deploy_sha","code":true},{"text":" on every log line is the detail most often missing and the one that most often resolves an incident, because it lets you filter to the version and answer the question above directly."}]},{"t":"p","runs":[{"text":"The three that matter most: a request identifier that travels across services, the inputs and outputs at every external boundary, and the version. Everything else is a refinement."}]},{"t":"h2","id":"runbooks-written-for-the-person-who-does-not-know","runs":[{"text":"Runbooks, written for the person who does not know"}]},{"t":"p","runs":[{"text":"A runbook assuming familiarity is useless at 3am to somebody who does not have it, which, increasingly, is whoever is on call."}]},{"t":"p","runs":[{"text":"One page per alert:"}]},{"t":"code","lang":"markdown","label":"Markdown","text":"# Alert: checkout_error_rate > 2%\n\nWhat it means: more than 2% of POST /checkout returned 5xx over 5 minutes.\nWho it affects: customers cannot complete purchases. Revenue-affecting.\n\nFirst: check https://dash/deploys — anything in the last hour?\n        If yes, roll back first and diagnose afterwards.\n\nThen:  check the provider status page (link).\n       Run: `kubectl logs -l app=checkout --since=10m | grep -c 'provider timeout'`\n       If provider timeouts dominate, set flag `checkout.fallback_provider=true`.\n\nEscalate to: #payments-oncall. Ravi knows the reconciliation path.\nDo NOT: re-run the settlement job. It is not idempotent. See ADR 0031."},{"t":"p","runs":[{"text":"The "},{"text":"do not","i":true},{"text":" section is the one people skip and the one that prevents the second, worse incident."}]},{"t":"h2","id":"roll-back-first","runs":[{"text":"Roll back first"}]},{"t":"p","runs":[{"text":"With more change flowing and less of it deeply understood, the balance shifted further towards reverting before diagnosing. Diagnosis takes as long as it takes; a rollback takes minutes and stops the bleeding."}]},{"t":"p","runs":[{"text":"The practices that make this available: deploy small, deploy often, keep the previous version ready, make migrations backwards compatible for at least one release, and keep flags for anything risky. These are the same practices that make everything else in this course work, which is not a coincidence: reversibility is the property that makes speed survivable at every level."}]},{"t":"h2","id":"the-review-and-the-one-output-that-matters","runs":[{"text":"The review, and the one output that matters"}]},{"t":"p","runs":[{"text":"Blameless, because a review that assigns fault produces careful reporting rather than accurate reporting, and you need the accurate kind."}]},{"t":"p","runs":[{"text":"One mechanical change worth making: the standard prompt has always been “what did the person do and why was it reasonable at the time?” Extend it to “what did the system present to the person, and why was that presentation reasonable to act on?” A generated change that looked correct, passed review, and was wrong is a systems failure with several contributing causes, and hunting for the individual who should have caught it finds nothing useful."}]},{"t":"p","runs":[{"text":"And the output that matters is not the document. It is "},{"text":"a permanent test.","b":true}]},{"t":"p","runs":[{"text":"Every incident produces one, in the same week, before anybody has moved on. A property, a database constraint, a monitor, an assertion, an item on the hostile-inputs corpus. The suite that catches real problems is built almost entirely out of things that actually happened, and a team that does this consistently has, after two years, a test suite worth more than anything anyone could have designed in advance."}]},{"t":"p","runs":[{"text":"An incident review with no test attached is a document. An incident review with a test attached is verification capacity, which is the constrained resource this entire course is about."}]}],"check":{"question":[{"text":"A team’s incident reviews are thorough, blameless and well written, yet the same class of failure recurs three times in a year. What is most likely missing?"}],"options":[{"text":[{"text":"Follow-up action items with owners and deadlines, since well-written reviews often produce recommendations nobody executes."}],"correct":false,"tempting":[{"text":"Close, and better than nothing, but an action item is a promise while a test is an enforcement that cannot be quietly dropped."}]},{"text":[{"text":"Each review produced a document but no permanent automated check."}],"correct":true,"tempting":null},{"text":[{"text":"The reviews are too blameless, so no individual felt responsible for preventing a recurrence."}],"correct":false,"tempting":[{"text":"Reintroduces blame as a control, which produces more careful reporting rather than fewer failures."}]},{"text":[{"text":"The root cause analysis stopped too early, identifying a proximate trigger rather than the underlying systemic condition."}],"correct":false,"tempting":[{"text":"A real weakness in shallow reviews, but the description says they are thorough: the gap is between understanding and enforcement."}]}],"order":[2,3,1,0],"why":null},"prev":{"slug":"onboarding-without-an-author","title":"Onboarding into code nobody wrote","minutes":8},"next":{"slug":"a-tool-policy-people-follow","title":"A policy people will actually follow","minutes":9}}}