{"course":{"slug":"cloud-and-shipping","title":"Shipping It","url":"https://addaly.com/learn/cloud-and-shipping","level":"intermediate","track":"build"},"lesson":{"slug":"slos-and-error-budgets","title":"Deciding how reliable it has to be","url":"https://addaly.com/learn/cloud-and-shipping/slos-and-error-budgets","position":27,"of":74,"minutes":10,"answer":"Pick a reliability target you can defend, express it as good events over valid events.","keyIdea":"Pick a reliability target you can defend, express it as good events over valid events, and treat the remaining failure allowance as a budget you spend on shipping: with burn-rate alerts so one rule catches both the outage and the slow bleed."},"body":{"format":"markdown","text":"## \"It should always work\" is not a target\n\nAsk anybody what reliability they want and the answer is 100%. It is the wrong answer, and not for the reason people expect.\n\n100% is unreachable, because your users reach you through networks, DNS resolvers, browsers and phones you do not control, and those fail more often than your service will. Long before your own uptime becomes the limiting factor, you are spending money to improve a number nobody can perceive. Every extra nine costs roughly ten times the previous one, and buys a difference the user cannot distinguish from their own wifi.\n\nSo the useful question is not \"how reliable can we be\" but \"how reliable do we need to be, and what are we willing to give up for it\". That number, written down, is a service level objective.\n\n## The three words, in order\n\n- **SLI: the indicator.** A measurement. \"The proportion of requests that return a non-5xx status within 500 ms.\"\n- **SLO: the objective.** A target for that measurement over a window. \"99.5% of requests, over 30 days.\"\n- **SLA: the agreement.** The same idea in a contract, with money attached when you miss it. Most small projects have no SLA and should not pretend otherwise.\n\nWrite the SLI as *good events divided by valid events*. That formulation forces you to define both halves, and the arguments happen at the right moment: before the incident, not during it. Are health checks valid events? Are requests from a bot? Is a 429 to an abusive client a failure? Decide once, write it down.\n\n## What the numbers mean in minutes\n\nOver 30 days, allowable downtime is:\n\n- 99%: 7 hours 18 minutes.\n- 99.5%: 3 hours 39 minutes.\n- 99.9%: 43 minutes.\n- 99.95%: 21 minutes.\n- 99.99%: 4 minutes 20 seconds.\n\nRead that last line and notice what it demands: nobody can be woken, diagnose, and fix anything in four minutes. Four nines is not an operations target, it is an architecture: automatic failover, no single points of failure, and a team on rotation. A one-person project promising four nines is promising something it structurally cannot deliver.\n\n> Figure: What each extra nine allows in a month. Bars (minutes of downtime allowed in 30 days): 99% 438; 99.5% 219; 99.9% 43; 99.95% 21; 99.99% 4.3. Nobody can be woken, diagnose and fix anything in four minutes. Four nines is not an operations target, it is an architecture with automatic failover and a team on rotation.\n\nFor most small sites, 99.5% or 99.9% is honest, and the first is easier to defend than people think.\n\n## The error budget is the useful half\n\nIf the objective is 99.9%, then 0.1% of requests are allowed to fail. That is not a shameful residue; it is a budget, and it is meant to be spent.\n\nThe budget converts reliability from an argument into arithmetic. Plenty of budget left this month means ship the risky change, run the migration, try the new provider. Budget exhausted means stop shipping features and spend the time on reliability instead. The rule is written down in advance, so the decision during a bad month is not a negotiation between the person who wants to ship and the person who was woken up.\n\nA small team can run this informally: one number on the dashboard, checked at the start of each week.\n\n## Burn rate, which is how you alert on it\n\nAlerting whenever the error rate is non-zero produces noise. Alerting when the budget is fully spent is too late. Burn rate sits between: it asks how fast you are consuming the budget relative to the whole window.\n\nA burn rate of 1 spends exactly the month's budget in a month. A burn rate of 14.4 spends it in about two days, and a common pairing is: page if you have burnt 2% of the budget in one hour, ticket if you have burnt 10% in six hours. The fast alert catches outages, the slow alert catches a steady drizzle that would otherwise never trip a threshold.\n\nThe practical benefit is that one alert covers both the sudden total outage and the slow bleed, and neither needs a hand-tuned latency threshold.\n\n## Measure it where the user is\n\nAn SLI computed from your own server logs cannot see the requests that never arrived: DNS failure, TLS expiry, a load balancer returning 502 on its own, the whole region unreachable. Those are exactly the worst outages.\n\nSo compute the indicator as close to the user as you can afford: the CDN or load balancer's logs rather than the application's, plus an external synthetic check from somewhere else entirely. Free tiers of uptime services do the second part; a cron job on a different provider's free instance does it for nothing.\n\n## The version for one person\n\nYou do not need a platform for this. You need:\n\n1. One SLI, defined in a sentence, in the repository.\n2. One target, with a window.\n3. One number on a dashboard: budget remaining this month.\n4. One rule about what happens when it runs out.\n\nThat is a working reliability practice. Everything else (multi-window multi-burn-rate alerting, per-endpoint objectives, formal reviews) is scaling that up, and it is worth adding only when the simple version has been used for a few months and found insufficient."},"check":{"question":"A service with a 99.9% monthly availability objective has burnt 60% of its error budget by day four, entirely in one 25-minute incident. Under a budget-based policy, what does that imply?","options":[{"text":"The objective was set too high and should be lowered to 99.5%","correct":false,"tempting":"Changing the target because you missed it removes the signal; the target is a product decision, not something adjusted to match the current month's luck."},{"text":"Nothing yet: the objective is measured over 30 days and the month is still within target","correct":false,"tempting":"True arithmetically and misses the point of a budget: the remaining 40% has to cover 26 more days, which is the decision the policy exists to force."},{"text":"Most of the month's allowance is gone, so risky changes should wait and reliability work should take priority","correct":true},{"text":"An alert should fire immediately, because 60% consumed exceeds any sensible burn-rate threshold","correct":false,"tempting":"Confuses the budget level with the burn rate; the burn happened during the incident and alerting has already served its purpose."}]},"prev":{"slug":"alerts-worth-waking-for","title":"Alerts you would get out of bed for","url":"https://addaly.com/learn/cloud-and-shipping/alerts-worth-waking-for"},"next":{"slug":"tracing-one-slow-request","title":"Finding the slow bit","url":"https://addaly.com/learn/cloud-and-shipping/tracing-one-slow-request"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Deciding how reliable it has to be', in Shipping It. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/cloud-and-shipping/slos-and-error-budgets (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Deciding how reliable it has to be","courseTitle":"Shipping It","module":{"index":3,"slug":"knowing-what-it-is-doing","title":"Knowing what it is doing"},"answer":null,"keyIdea":"Pick a reliability target you can defend, express it as good events over valid events, and treat the remaining failure allowance as a budget you spend on shipping: with burn-rate alerts so one rule catches both the outage and the slow bleed.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"minutes of downtime allowed in 30 days","title":"What each extra nine allows in a month","series":[{"label":"99%","value":438},{"label":"99.5%","value":219,"highlight":true},{"label":"99.9%","value":43},{"label":"99.95%","value":21},{"label":"99.99%","value":4.3}],"caption":"Nobody can be woken, diagnose and fix anything in four minutes. Four nines is not an operations target, it is an architecture with automatic failover and a team on rotation."},"alt":"Figure: What each extra nine allows in a month. Bars (minutes of downtime allowed in 30 days): 99% 438; 99.5% 219; 99.9% 43; 99.95% 21; 99.99% 4.3. Nobody can be woken, diagnose and fix anything in four minutes. Four nines is not an operations target, it is an architecture with automatic failover and a team on rotation."}},"blocks":[{"t":"h2","id":"it-should-always-work-is-not-a-target","runs":[{"text":"“It should always work” is not a target"}]},{"t":"p","runs":[{"text":"Ask anybody what reliability they want and the answer is 100%. It is the wrong answer, and not for the reason people expect."}]},{"t":"p","runs":[{"text":"100% is unreachable, because your users reach you through networks, DNS resolvers, browsers and phones you do not control, and those fail more often than your service will. Long before your own uptime becomes the limiting factor, you are spending money to improve a number nobody can perceive. Every extra nine costs roughly ten times the previous one, and buys a difference the user cannot distinguish from their own wifi."}]},{"t":"p","runs":[{"text":"So the useful question is not “how reliable can we be” but “how reliable do we need to be, and what are we willing to give up for it”. That number, written down, is a service level objective."}]},{"t":"h2","id":"the-three-words-in-order","runs":[{"text":"The three words, in order"}]},{"t":"ul","items":[[{"text":"SLI: the indicator.","b":true},{"text":" A measurement. “The proportion of requests that return a non-5xx status within 500 ms.”"}],[{"text":"SLO: the objective.","b":true},{"text":" A target for that measurement over a window. “99.5% of requests, over 30 days.”"}],[{"text":"SLA: the agreement.","b":true},{"text":" The same idea in a contract, with money attached when you miss it. Most small projects have no SLA and should not pretend otherwise."}]]},{"t":"p","runs":[{"text":"Write the SLI as "},{"text":"good events divided by valid events","i":true},{"text":". That formulation forces you to define both halves, and the arguments happen at the right moment: before the incident, not during it. Are health checks valid events? Are requests from a bot? Is a 429 to an abusive client a failure? Decide once, write it down."}]},{"t":"h2","id":"what-the-numbers-mean-in-minutes","runs":[{"text":"What the numbers mean in minutes"}]},{"t":"p","runs":[{"text":"Over 30 days, allowable downtime is:"}]},{"t":"ul","items":[[{"text":"99%: 7 hours 18 minutes."}],[{"text":"99.5%: 3 hours 39 minutes."}],[{"text":"99.9%: 43 minutes."}],[{"text":"99.95%: 21 minutes."}],[{"text":"99.99%: 4 minutes 20 seconds."}]]},{"t":"p","runs":[{"text":"Read that last line and notice what it demands: nobody can be woken, diagnose, and fix anything in four minutes. Four nines is not an operations target, it is an architecture: automatic failover, no single points of failure, and a team on rotation. A one-person project promising four nines is promising something it structurally cannot deliver."}]},{"t":"p","runs":[{"text":"For most small sites, 99.5% or 99.9% is honest, and the first is easier to defend than people think."}]},{"t":"h2","id":"the-error-budget-is-the-useful-half","runs":[{"text":"The error budget is the useful half"}]},{"t":"p","runs":[{"text":"If the objective is 99.9%, then 0.1% of requests are allowed to fail. That is not a shameful residue; it is a budget, and it is meant to be spent."}]},{"t":"p","runs":[{"text":"The budget converts reliability from an argument into arithmetic. Plenty of budget left this month means ship the risky change, run the migration, try the new provider. Budget exhausted means stop shipping features and spend the time on reliability instead. The rule is written down in advance, so the decision during a bad month is not a negotiation between the person who wants to ship and the person who was woken up."}]},{"t":"p","runs":[{"text":"A small team can run this informally: one number on the dashboard, checked at the start of each week."}]},{"t":"h2","id":"burn-rate-which-is-how-you-alert-on-it","runs":[{"text":"Burn rate, which is how you alert on it"}]},{"t":"p","runs":[{"text":"Alerting whenever the error rate is non-zero produces noise. Alerting when the budget is fully spent is too late. Burn rate sits between: it asks how fast you are consuming the budget relative to the whole window."}]},{"t":"p","runs":[{"text":"A burn rate of 1 spends exactly the month’s budget in a month. A burn rate of 14.4 spends it in about two days, and a common pairing is: page if you have burnt 2% of the budget in one hour, ticket if you have burnt 10% in six hours. The fast alert catches outages, the slow alert catches a steady drizzle that would otherwise never trip a threshold."}]},{"t":"p","runs":[{"text":"The practical benefit is that one alert covers both the sudden total outage and the slow bleed, and neither needs a hand-tuned latency threshold."}]},{"t":"h2","id":"measure-it-where-the-user-is","runs":[{"text":"Measure it where the user is"}]},{"t":"p","runs":[{"text":"An SLI computed from your own server logs cannot see the requests that never arrived: DNS failure, TLS expiry, a load balancer returning 502 on its own, the whole region unreachable. Those are exactly the worst outages."}]},{"t":"p","runs":[{"text":"So compute the indicator as close to the user as you can afford: the CDN or load balancer’s logs rather than the application’s, plus an external synthetic check from somewhere else entirely. Free tiers of uptime services do the second part; a cron job on a different provider’s free instance does it for nothing."}]},{"t":"h2","id":"the-version-for-one-person","runs":[{"text":"The version for one person"}]},{"t":"p","runs":[{"text":"You do not need a platform for this. You need:"}]},{"t":"ol","start":1,"items":[[{"text":"One SLI, defined in a sentence, in the repository."}],[{"text":"One target, with a window."}],[{"text":"One number on a dashboard: budget remaining this month."}],[{"text":"One rule about what happens when it runs out."}]]},{"t":"p","runs":[{"text":"That is a working reliability practice. Everything else (multi-window multi-burn-rate alerting, per-endpoint objectives, formal reviews) is scaling that up, and it is worth adding only when the simple version has been used for a few months and found insufficient."}]}],"check":{"question":[{"text":"A service with a 99.9% monthly availability objective has burnt 60% of its error budget by day four, entirely in one 25-minute incident. Under a budget-based policy, what does that imply?"}],"options":[{"text":[{"text":"The objective was set too high and should be lowered to 99.5%"}],"correct":false,"tempting":[{"text":"Changing the target because you missed it removes the signal; the target is a product decision, not something adjusted to match the current month’s luck."}]},{"text":[{"text":"Nothing yet: the objective is measured over 30 days and the month is still within target"}],"correct":false,"tempting":[{"text":"True arithmetically and misses the point of a budget: the remaining 40% has to cover 26 more days, which is the decision the policy exists to force."}]},{"text":[{"text":"Most of the month’s allowance is gone, so risky changes should wait and reliability work should take priority"}],"correct":true,"tempting":null},{"text":[{"text":"An alert should fire immediately, because 60% consumed exceeds any sensible burn-rate threshold"}],"correct":false,"tempting":[{"text":"Confuses the budget level with the burn rate; the burn happened during the incident and alerting has already served its purpose."}]}],"order":[1,3,2,0],"why":null},"prev":{"slug":"alerts-worth-waking-for","title":"Alerts you would get out of bed for","minutes":8},"next":{"slug":"tracing-one-slow-request","title":"Finding the slow bit","minutes":10}}}