{"course":{"slug":"cloud-and-shipping","title":"Shipping It","url":"https://addaly.com/learn/cloud-and-shipping","level":"intermediate","track":"build"},"lesson":{"slug":"alerts-worth-waking-for","title":"Alerts you would get out of bed for","url":"https://addaly.com/learn/cloud-and-shipping/alerts-worth-waking-for","position":26,"of":74,"minutes":8,"answer":"Every alert must pass two tests (a human is needed, and needed now) and an alert that fails either one is training you to ignore the ones that do not.","keyIdea":"Every alert must pass two tests (a human is needed, and needed now) and an alert that fails either one is training you to ignore the ones that do not."},"body":{"format":"markdown","text":"## Two questions, asked before you create any alert\n\n1. **Does this need a human?** If the system recovers on its own, or the fix is automatic, it does not.\n2. **Does it need one now?** If the honest answer is \"I would look at this on Tuesday\", it is a ticket or a dashboard, not a page.\n\nAn alert that fails either test is not merely useless. It is actively harmful, because it spends your attention and teaches you that alerts can be ignored. That lesson generalises to the alert that mattered.\n\n> Figure: Two questions before any alert exists. A two by two grid, rows Needs a human and Recovers on its own, columns Needed now and Can wait: Needs a human and Needed now: Page; Needs a human and Can wait: Ticket; Recovers on its own and Needed now: Dashboard panel; Recovers on its own and Can wait: Nothing. Only the top-left box earns a page. Everything else is a ticket, a dashboard panel or nothing, and an alert that lands elsewhere trains you to ignore the one that belongs there.\n\n## Alert on symptoms, not causes\n\n- **Cause alert:** CPU is above 90%.\n- **Symptom alert:** more than 5% of requests are returning 5xx.\n\nCPU at 90% with every user happy is a well-utilised machine and a fine Tuesday. CPU at 30% while the connection pool is exhausted is an outage. The resource number tells you nothing on its own; the user-facing number always means something.\n\nSo page on the small set of things a user would notice:\n\n- The site is unreachable from outside.\n- The error rate crossed a threshold and stayed there.\n- Latency at p99 is several times normal for more than a few minutes.\n- A queue is growing and not draining.\n- Certificate or domain expiry is close.\n\nKeep the cause metrics on a dashboard. They are what you look at *after* the page, to work out why.\n\n## Sustained, not instantaneous\n\nAlmost every false page is a threshold crossed for one sample. Require the condition to hold for a few minutes before it fires, and require it to clear for a few minutes before it resolves; otherwise you get a flapping alert, which is three pages instead of one and no extra information.\n\n## Burn rate, if you want to be precise about it\n\nIf you have set a target, say 99.9% of requests succeed, that is an error budget of about 43 minutes a month. You can then alert on how fast you are consuming it rather than on a raw error rate.\n\nBurning budget at 14 times the sustainable rate over an hour means you will exhaust the month in about two days: page. Burning at 3 times over six hours means something is wrong but it can wait for the morning: ticket.\n\nThe value of this framing is that it makes the trade-off explicit. Any threshold implies a tolerance; burn rate just makes you write yours down.\n\n## Every alert needs a runbook link and an owner\n\nThe alert body should answer: what is broken, how bad, and what do I do first. A page that says `HighErrorRate: firing` at 3am and nothing else means the responder spends the first ten minutes reconstructing context they could have been handed.\n\nIf an alert has no runbook entry, either write one or delete the alert. There is no third option that survives contact with a real night.\n\n## The number that tells you the system is broken\n\nTrack what proportion of pages result in action. If more than a fifth are being acknowledged and ignored, the alerting system is decoration and you are being trained out of responding. The fix is to delete alerts, which feels irresponsible and is the correct move. Fewer, better alerts beat comprehensive coverage nobody reads.\n\n## The version for a small project\n\nIf this is a side project or a one-person product, most 3am pages are wrong. You are not going to fix a subtle data bug at 3am, and trying will cost you the whole next day.\n\nSo decide out loud, and write it in the runbook:\n\n- **Wake me:** the site is completely down, or money is being lost or spent, or data is being corrupted.\n- **Tell me in the morning:** everything else.\n\nThere is no shame in that list. It is a better on-call policy than an aspirational one that you will abandon in the third week.\n\n## The free path\n\nGrafana's built-in alerting is free and self-hosted, and Uptime Kuma alerts on its own checks. For delivery to a phone: **ntfy.sh** is open source, self-hostable, and pushes to your phone with a curl command. A Telegram bot works. So does a phone alarm triggered by a webhook.\n\nPagerDuty and Opsgenie sell escalation policies, schedules and the guarantee that somebody eventually wakes: worth real money for a team with a rota, worth nothing for one person with a phone."},"check":{"question":"You have twelve alerts. Nine of them fire at least weekly, and the team has learned which ones can be dismissed without looking. What should you do first?","options":[{"text":"Add severity levels so the important alerts are visually distinct from the noisy ones","correct":false,"tempting":"Puts a label on the problem rather than removing it; people who have learned to dismiss alerts will learn to dismiss a severity too."},{"text":"Delete or downgrade the alerts that do not require a human right now, even though it feels like reducing coverage","correct":true},{"text":"Route the noisy alerts to a separate channel so the critical channel stays clean","correct":false,"tempting":"Better than nothing, and a reasonable second step, but the noisy channel becomes unread, so those conditions are now unmonitored while appearing to be covered."},{"text":"Raise the thresholds on the nine noisy alerts until they stop firing weekly","correct":false,"tempting":"Treats frequency as the defect. Some of those conditions genuinely occur weekly and simply do not need a human; a higher threshold hides them without answering the question."}]},"prev":{"slug":"health-checks-that-tell-the-truth","title":"The health check that lies","url":"https://addaly.com/learn/cloud-and-shipping/health-checks-that-tell-the-truth"},"next":{"slug":"slos-and-error-budgets","title":"Deciding how reliable it has to be","url":"https://addaly.com/learn/cloud-and-shipping/slos-and-error-budgets"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Alerts you would get out of bed for', in Shipping It. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/cloud-and-shipping/alerts-worth-waking-for (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Alerts you would get out of bed for","courseTitle":"Shipping It","module":{"index":3,"slug":"knowing-what-it-is-doing","title":"Knowing what it is doing"},"answer":null,"keyIdea":"Every alert must pass two tests (a human is needed, and needed now) and an alert that fails either one is training you to ignore the ones that do not.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["Needed now","Can wait"],"rows":["Needs a human","Recovers on its own"],"type":"matrix","cells":[["Page","Ticket"],["Dashboard panel","Nothing"]],"title":"Two questions before any alert exists","caption":"Only the top-left box earns a page. Everything else is a ticket, a dashboard panel or nothing, and an alert that lands elsewhere trains you to ignore the one that belongs there.","cellLabels":[["wake somebody","morning"],["look after the page","delete the rule"]]},"alt":"Figure: Two questions before any alert exists. A two by two grid, rows Needs a human and Recovers on its own, columns Needed now and Can wait: Needs a human and Needed now: Page; Needs a human and Can wait: Ticket; Recovers on its own and Needed now: Dashboard panel; Recovers on its own and Can wait: Nothing. Only the top-left box earns a page. Everything else is a ticket, a dashboard panel or nothing, and an alert that lands elsewhere trains you to ignore the one that belongs there."}},"blocks":[{"t":"h2","id":"two-questions-asked-before-you-create-any-alert","runs":[{"text":"Two questions, asked before you create any alert"}]},{"t":"ol","start":1,"items":[[{"text":"Does this need a human?","b":true},{"text":" If the system recovers on its own, or the fix is automatic, it does not."}],[{"text":"Does it need one now?","b":true},{"text":" If the honest answer is “I would look at this on Tuesday”, it is a ticket or a dashboard, not a page."}]]},{"t":"p","runs":[{"text":"An alert that fails either test is not merely useless. It is actively harmful, because it spends your attention and teaches you that alerts can be ignored. That lesson generalises to the alert that mattered."}]},{"t":"h2","id":"alert-on-symptoms-not-causes","runs":[{"text":"Alert on symptoms, not causes"}]},{"t":"ul","items":[[{"text":"Cause alert:","b":true},{"text":" CPU is above 90%."}],[{"text":"Symptom alert:","b":true},{"text":" more than 5% of requests are returning 5xx."}]]},{"t":"p","runs":[{"text":"CPU at 90% with every user happy is a well-utilised machine and a fine Tuesday. CPU at 30% while the connection pool is exhausted is an outage. The resource number tells you nothing on its own; the user-facing number always means something."}]},{"t":"p","runs":[{"text":"So page on the small set of things a user would notice:"}]},{"t":"ul","items":[[{"text":"The site is unreachable from outside."}],[{"text":"The error rate crossed a threshold and stayed there."}],[{"text":"Latency at p99 is several times normal for more than a few minutes."}],[{"text":"A queue is growing and not draining."}],[{"text":"Certificate or domain expiry is close."}]]},{"t":"p","runs":[{"text":"Keep the cause metrics on a dashboard. They are what you look at "},{"text":"after","i":true},{"text":" the page, to work out why."}]},{"t":"h2","id":"sustained-not-instantaneous","runs":[{"text":"Sustained, not instantaneous"}]},{"t":"p","runs":[{"text":"Almost every false page is a threshold crossed for one sample. Require the condition to hold for a few minutes before it fires, and require it to clear for a few minutes before it resolves; otherwise you get a flapping alert, which is three pages instead of one and no extra information."}]},{"t":"h2","id":"burn-rate-if-you-want-to-be-precise-about-it","runs":[{"text":"Burn rate, if you want to be precise about it"}]},{"t":"p","runs":[{"text":"If you have set a target, say 99.9% of requests succeed, that is an error budget of about 43 minutes a month. You can then alert on how fast you are consuming it rather than on a raw error rate."}]},{"t":"p","runs":[{"text":"Burning budget at 14 times the sustainable rate over an hour means you will exhaust the month in about two days: page. Burning at 3 times over six hours means something is wrong but it can wait for the morning: ticket."}]},{"t":"p","runs":[{"text":"The value of this framing is that it makes the trade-off explicit. Any threshold implies a tolerance; burn rate just makes you write yours down."}]},{"t":"h2","id":"every-alert-needs-a-runbook-link-and-an-owner","runs":[{"text":"Every alert needs a runbook link and an owner"}]},{"t":"p","runs":[{"text":"The alert body should answer: what is broken, how bad, and what do I do first. A page that says "},{"text":"HighErrorRate: firing","code":true},{"text":" at 3am and nothing else means the responder spends the first ten minutes reconstructing context they could have been handed."}]},{"t":"p","runs":[{"text":"If an alert has no runbook entry, either write one or delete the alert. There is no third option that survives contact with a real night."}]},{"t":"h2","id":"the-number-that-tells-you-the-system-is-broken","runs":[{"text":"The number that tells you the system is broken"}]},{"t":"p","runs":[{"text":"Track what proportion of pages result in action. If more than a fifth are being acknowledged and ignored, the alerting system is decoration and you are being trained out of responding. The fix is to delete alerts, which feels irresponsible and is the correct move. Fewer, better alerts beat comprehensive coverage nobody reads."}]},{"t":"h2","id":"the-version-for-a-small-project","runs":[{"text":"The version for a small project"}]},{"t":"p","runs":[{"text":"If this is a side project or a one-person product, most 3am pages are wrong. You are not going to fix a subtle data bug at 3am, and trying will cost you the whole next day."}]},{"t":"p","runs":[{"text":"So decide out loud, and write it in the runbook:"}]},{"t":"ul","items":[[{"text":"Wake me:","b":true},{"text":" the site is completely down, or money is being lost or spent, or data is being corrupted."}],[{"text":"Tell me in the morning:","b":true},{"text":" everything else."}]]},{"t":"p","runs":[{"text":"There is no shame in that list. It is a better on-call policy than an aspirational one that you will abandon in the third week."}]},{"t":"h2","id":"the-free-path","runs":[{"text":"The free path"}],"role":"free"},{"t":"p","runs":[{"text":"Grafana’s built-in alerting is free and self-hosted, and Uptime Kuma alerts on its own checks. For delivery to a phone: "},{"text":"ntfy.sh","b":true},{"text":" is open source, self-hostable, and pushes to your phone with a curl command. A Telegram bot works. So does a phone alarm triggered by a webhook."}]},{"t":"p","runs":[{"text":"PagerDuty and Opsgenie sell escalation policies, schedules and the guarantee that somebody eventually wakes: worth real money for a team with a rota, worth nothing for one person with a phone."}]}],"check":{"question":[{"text":"You have twelve alerts. Nine of them fire at least weekly, and the team has learned which ones can be dismissed without looking. What should you do first?"}],"options":[{"text":[{"text":"Add severity levels so the important alerts are visually distinct from the noisy ones"}],"correct":false,"tempting":[{"text":"Puts a label on the problem rather than removing it; people who have learned to dismiss alerts will learn to dismiss a severity too."}]},{"text":[{"text":"Delete or downgrade the alerts that do not require a human right now, even though it feels like reducing coverage"}],"correct":true,"tempting":null},{"text":[{"text":"Route the noisy alerts to a separate channel so the critical channel stays clean"}],"correct":false,"tempting":[{"text":"Better than nothing, and a reasonable second step, but the noisy channel becomes unread, so those conditions are now unmonitored while appearing to be covered."}]},{"text":[{"text":"Raise the thresholds on the nine noisy alerts until they stop firing weekly"}],"correct":false,"tempting":[{"text":"Treats frequency as the defect. Some of those conditions genuinely occur weekly and simply do not need a human; a higher threshold hides them without answering the question."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"health-checks-that-tell-the-truth","title":"The health check that lies","minutes":9},"next":{"slug":"slos-and-error-budgets","title":"Deciding how reliable it has to be","minutes":10}}}