{"course":{"slug":"agents-and-automation","title":"Agents and Automation","url":"https://addaly.com/learn/agents-and-automation","level":"intermediate","track":"build"},"lesson":{"slug":"alerts-and-the-runbook","title":"The 3am version","url":"https://addaly.com/learn/agents-and-automation/alerts-and-the-runbook","position":62,"of":72,"minutes":8,"answer":"Alert on the job not running and on outputs that look wrong, not on exceptions, and keep a six-line runbook next to the code.","keyIdea":"Alert on the job not running and on outputs that look wrong, not on exceptions, and keep a six-line runbook next to the code."},"body":{"format":"markdown","text":"## Alert on outcomes, not exceptions\n\n\"An exception occurred\" is noise. Within a fortnight everybody filters the channel and the alerts have become decoration.\n\nAlert on things that mean something:\n\n> Figure: Two alerting policies, a fortnight apart. What everybody filters within a fortnight: An exception occurred; A single HTTP 500 from the shipping API; The model returned invalid JSON, once; A retry happened. What is worth telling somebody: The job did not run at all; It usually writes four hundred rows and wrote three; The error rate over the last hour crossed the threshold; Queue depth is growing, so you are falling behind rather than failing; Spend crossed half the cap, and then the cap; The validator is rejecting more than its usual share of outputs. A job that has died produces no errors, because it is not running, so every error-based alert is blind to the most common serious failure. The fix is a heartbeat: the job pings on success and something else notices the ping is late.\n\n- **The job did not run.** More on this below; it is the one people miss.\n- **The output is outside its normal range.** It usually produces 400 rows and produced 3. Or 40,000.\n- **The error rate over the last hour crossed a threshold**, not a single error.\n- **The queue depth is growing**, meaning you are falling behind rather than failing.\n- **Spend crossed a line**, at half the cap and again at the cap.\n- **The validator rejected more than the usual share** of outputs. This is your early warning that a model or an upstream format has changed.\n\n## The heartbeat, which is the alert nobody has\n\nA job that dies produces no errors, because it is not running. Every error-based alerting system is blind to the most common serious failure.\n\nThe fix is a dead man's switch: the job pings a URL on success, and if the ping does not arrive within the expected window, *you* get alerted. Healthchecks.io has a free tier and is self-hostable; Cronitor and others do the same. The homemade version is a `last_success_at` row and a second, much simpler job that checks it, and yes, you then need a heartbeat on the checker, which is why most people use the hosted one for that outer layer.\n\n## A dead letter queue\n\nItems that fail after all retries go into a table with the payload, the error and the timestamp. Not to a log file, which expires and which nobody reads.\n\nThen somebody can look at what accumulated, fix the cause, and replay the queue. Without one, failures are permanent and invisible, and you find out about them from a customer.\n\n## Degrade rather than stop\n\nWhen a dependency is down, an automation has better options than dying:\n\n- queue the work and retry when it returns\n- fall back to a smaller or different model\n- fall back to the deterministic path, and mark the output so downstream knows it was not enriched\n- deliver a partial result, clearly labelled\n\nEach is a decision you make now, in daylight, and write into the code. None of them can be improvised at 3am.\n\n## The runbook: six lines, in the repository\n\nNot a wiki page. A `RUNBOOK.md` beside the code, containing:\n\n1. **What it does**, in one sentence.\n2. **What it touches**: which systems, which credentials, what it can change.\n3. **How to stop it**, exactly. The command or the flag.\n4. **How to see the last runs** and their outcomes. A command or a link.\n5. **The three failures it has actually had**, and what fixed each.\n6. **Who owns it**, and who deputises.\n\nPoint five is the one that matters and the one that is always missing. It turns your incident history into somebody else's fifteen-minute fix, and it takes two minutes to add each time.\n\n## Practise the recovery\n\nOnce a quarter: stop it deliberately, or break it in a controlled way, and have somebody who did not build it restore it using only the runbook.\n\nYou will find that step 3 is out of date, that the link in step 4 has moved, and that the owner left in June. That is the point. A recovery procedure nobody has walked through is a hypothesis."},"check":{"question":"An hourly enrichment job silently stops running after a server reboot. Error alerting is configured and fires correctly on exceptions. Nobody notices for six days. What was missing?","options":[{"text":"Log aggregation, so the absence of log lines would have been visible.","correct":false,"tempting":"Would let somebody notice if they went looking, which is exactly what nobody does for six days; absence of logs is not an event."},{"text":"A heartbeat alert that fires when an expected successful run does not arrive, since a job that is not running produces no exceptions to alert on.","correct":true},{"text":"A retry policy, so the job would have restarted itself after the reboot.","correct":false,"tempting":"Retries apply to a run that failed, not to a scheduler that never started one; and a job that cannot start cannot retry."},{"text":"A lower alerting threshold on the existing error alerts.","correct":false,"tempting":"Tunes the sensitivity of a signal that had a value of zero for six days."}]},"prev":{"slug":"rollout-and-rollback","title":"Shadow, canary, and the off switch","url":"https://addaly.com/learn/agents-and-automation/rollout-and-rollback"},"next":{"slug":"the-maintenance-bill","title":"What it costs to keep it alive","url":"https://addaly.com/learn/agents-and-automation/the-maintenance-bill"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The 3am version', in Agents and Automation. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/agents-and-automation/alerts-and-the-runbook (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The 3am version","courseTitle":"Agents and Automation","module":{"index":7,"slug":"shipping-and-operating","title":"Shipping it, and keeping it alive"},"answer":null,"keyIdea":"Alert on the job not running and on outputs that look wrong, not on exceptions, and keep a six-line runbook next to the code.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"What everybody filters within a fortnight","items":["An exception occurred","A single HTTP 500 from the shipping API","The model returned invalid JSON, once","A retry happened"]},"type":"compare","right":{"title":"What is worth telling somebody","items":["The job did not run at all","It usually writes four hundred rows and wrote three","The error rate over the last hour crossed the threshold","Queue depth is growing, so you are falling behind rather than failing","Spend crossed half the cap, and then the cap","The validator is rejecting more than its usual share of outputs"]},"title":"Two alerting policies, a fortnight apart","caption":"A job that has died produces no errors, because it is not running, so every error-based alert is blind to the most common serious failure. The fix is a heartbeat: the job pings on success and something else notices the ping is late."},"alt":"Figure: Two alerting policies, a fortnight apart. What everybody filters within a fortnight: An exception occurred; A single HTTP 500 from the shipping API; The model returned invalid JSON, once; A retry happened. What is worth telling somebody: The job did not run at all; It usually writes four hundred rows and wrote three; The error rate over the last hour crossed the threshold; Queue depth is growing, so you are falling behind rather than failing; Spend crossed half the cap, and then the cap; The validator is rejecting more than its usual share of outputs. A job that has died produces no errors, because it is not running, so every error-based alert is blind to the most common serious failure. The fix is a heartbeat: the job pings on success and something else notices the ping is late."}},"blocks":[{"t":"h2","id":"alert-on-outcomes-not-exceptions","runs":[{"text":"Alert on outcomes, not exceptions"}]},{"t":"p","runs":[{"text":"“An exception occurred” is noise. Within a fortnight everybody filters the channel and the alerts have become decoration."}]},{"t":"p","runs":[{"text":"Alert on things that mean something:"}]},{"t":"ul","items":[[{"text":"The job did not run.","b":true},{"text":" More on this below; it is the one people miss."}],[{"text":"The output is outside its normal range.","b":true},{"text":" It usually produces 400 rows and produced 3. Or 40,000."}],[{"text":"The error rate over the last hour crossed a threshold","b":true},{"text":", not a single error."}],[{"text":"The queue depth is growing","b":true},{"text":", meaning you are falling behind rather than failing."}],[{"text":"Spend crossed a line","b":true},{"text":", at half the cap and again at the cap."}],[{"text":"The validator rejected more than the usual share","b":true},{"text":" of outputs. This is your early warning that a model or an upstream format has changed."}]]},{"t":"h2","id":"the-heartbeat-which-is-the-alert-nobody-has","runs":[{"text":"The heartbeat, which is the alert nobody has"}]},{"t":"p","runs":[{"text":"A job that dies produces no errors, because it is not running. Every error-based alerting system is blind to the most common serious failure."}]},{"t":"p","runs":[{"text":"The fix is a dead man’s switch: the job pings a URL on success, and if the ping does not arrive within the expected window, "},{"text":"you","i":true},{"text":" get alerted. Healthchecks.io has a free tier and is self-hostable; Cronitor and others do the same. The homemade version is a "},{"text":"last_success_at","code":true},{"text":" row and a second, much simpler job that checks it, and yes, you then need a heartbeat on the checker, which is why most people use the hosted one for that outer layer."}]},{"t":"h2","id":"a-dead-letter-queue","runs":[{"text":"A dead letter queue"}]},{"t":"p","runs":[{"text":"Items that fail after all retries go into a table with the payload, the error and the timestamp. Not to a log file, which expires and which nobody reads."}]},{"t":"p","runs":[{"text":"Then somebody can look at what accumulated, fix the cause, and replay the queue. Without one, failures are permanent and invisible, and you find out about them from a customer."}]},{"t":"h2","id":"degrade-rather-than-stop","runs":[{"text":"Degrade rather than stop"}]},{"t":"p","runs":[{"text":"When a dependency is down, an automation has better options than dying:"}]},{"t":"ul","items":[[{"text":"queue the work and retry when it returns"}],[{"text":"fall back to a smaller or different model"}],[{"text":"fall back to the deterministic path, and mark the output so downstream knows it was not enriched"}],[{"text":"deliver a partial result, clearly labelled"}]]},{"t":"p","runs":[{"text":"Each is a decision you make now, in daylight, and write into the code. None of them can be improvised at 3am."}]},{"t":"h2","id":"the-runbook-six-lines-in-the-repository","runs":[{"text":"The runbook: six lines, in the repository"}]},{"t":"p","runs":[{"text":"Not a wiki page. A "},{"text":"RUNBOOK.md","code":true},{"text":" beside the code, containing:"}]},{"t":"ol","start":1,"items":[[{"text":"What it does","b":true},{"text":", in one sentence."}],[{"text":"What it touches","b":true},{"text":": which systems, which credentials, what it can change."}],[{"text":"How to stop it","b":true},{"text":", exactly. The command or the flag."}],[{"text":"How to see the last runs","b":true},{"text":" and their outcomes. A command or a link."}],[{"text":"The three failures it has actually had","b":true},{"text":", and what fixed each."}],[{"text":"Who owns it","b":true},{"text":", and who deputises."}]]},{"t":"p","runs":[{"text":"Point five is the one that matters and the one that is always missing. It turns your incident history into somebody else’s fifteen-minute fix, and it takes two minutes to add each time."}]},{"t":"h2","id":"practise-the-recovery","runs":[{"text":"Practise the recovery"}]},{"t":"p","runs":[{"text":"Once a quarter: stop it deliberately, or break it in a controlled way, and have somebody who did not build it restore it using only the runbook."}]},{"t":"p","runs":[{"text":"You will find that step 3 is out of date, that the link in step 4 has moved, and that the owner left in June. That is the point. A recovery procedure nobody has walked through is a hypothesis."}]}],"check":{"question":[{"text":"An hourly enrichment job silently stops running after a server reboot. Error alerting is configured and fires correctly on exceptions. Nobody notices for six days. What was missing?"}],"options":[{"text":[{"text":"Log aggregation, so the absence of log lines would have been visible."}],"correct":false,"tempting":[{"text":"Would let somebody notice if they went looking, which is exactly what nobody does for six days; absence of logs is not an event."}]},{"text":[{"text":"A heartbeat alert that fires when an expected successful run does not arrive, since a job that is not running produces no exceptions to alert on."}],"correct":true,"tempting":null},{"text":[{"text":"A retry policy, so the job would have restarted itself after the reboot."}],"correct":false,"tempting":[{"text":"Retries apply to a run that failed, not to a scheduler that never started one; and a job that cannot start cannot retry."}]},{"text":[{"text":"A lower alerting threshold on the existing error alerts."}],"correct":false,"tempting":[{"text":"Tunes the sensitivity of a signal that had a value of zero for six days."}]}],"order":[0,1,3,2],"why":null},"prev":{"slug":"rollout-and-rollback","title":"Shadow, canary, and the off switch","minutes":9},"next":{"slug":"the-maintenance-bill","title":"What it costs to keep it alive","minutes":8}}}