{"course":{"slug":"cloud-and-shipping","title":"Shipping It","url":"https://addaly.com/learn/cloud-and-shipping","level":"intermediate","track":"build"},"lesson":{"slug":"when-your-provider-goes-down","title":"The outage that is not yours","url":"https://addaly.com/learn/cloud-and-shipping/when-your-provider-goes-down","position":44,"of":74,"minutes":8,"answer":"When the broken thing belongs to somebody else, your work is to prove it quickly, degrade or serve stale rather than wait.","keyIdea":"When the broken thing belongs to somebody else, your work is to prove it quickly, degrade or serve stale rather than wait, and communicate from infrastructure that does not share the failure. And most small projects should buy cheap redundancy for stateless providers rather than an architecture they cannot operate."},"body":{"format":"markdown","text":"## Sooner or later it is not your fault\n\nAt some point the thing that is broken is a certificate authority, a DNS provider, a cloud region, a payment gateway or an identity provider, and nothing in your repository can fix it. This has happened at industry scale several times: a configuration change at a single CDN taking a large share of the web offline for an hour, a DNS provider under attack making dozens of well-known services unreachable, a cloud region failing and taking down the products built on it, including, memorably, some of the status pages meant to report the failure.\n\nYour job during those hours is not to fix somebody else's system. It is to know quickly that it is theirs, to keep as much of your product working as you can, and to tell your users something true.\n\n## Establishing that it is them, in five minutes\n\nA rough order that usually settles it:\n\n1. **Check from outside your own network.** A phone on mobile data, or a free external checker. Half of \"the site is down\" reports are one office's wifi or one ISP's DNS resolver.\n2. **Check the dependency directly.** `curl` its health endpoint or an unauthenticated URL from your server. Time it.\n3. **Check their status page and their social feed.** Status pages lag, often by twenty minutes, because a human has to publish. The social feed is usually faster, and third-party outage trackers faster still.\n4. **Check your own dependency dashboard**: the second screen from the dashboards lesson exists for precisely this moment.\n\nWrite the answer down with a timestamp as you go. In the postmortem, \"we knew it was the provider at 14:07\" is a much better sentence than \"we think it was maybe an hour\".\n\n## What you can do, ranked by how much it actually helps\n\n- **Degrade the feature, keep the site.** The failing-well lesson's flag per dependency pays for itself here. Turning off search or recommendations is a ten-second action that keeps everything else alive.\n- **Queue the work.** If the dependency is an email or webhook target, buffer and retry later. Users do not need those to be synchronous, and they rarely notice a twenty-minute delay.\n- **Serve stale.** A cache configured with `stale-if-error` will keep answering from an old copy while the origin or an upstream is failing. This one line of configuration converts a total outage into slightly out-of-date content.\n- **Switch provider, if you already can.** A second mail provider or model provider behind one interface is a cheap redundancy, because both are stateless HTTP APIs. This is realistic for mail, SMS, models and image processing.\n- **Wait, and communicate.** For DNS, identity or your database's region, there is often no alternative and pretending otherwise wastes the hour.\n\n## What is worth being redundant, and what is not\n\nAn honest ranking for a small project:\n\n**Worth it, because it is cheap:** a second DNS provider (the protocol supports multiple authoritative servers), a second mail sender, a second model or API provider behind your own interface, backups stored at a different company from the database.\n\n**Not worth it for most:** multi-region active-active, multi-cloud, a standby database on another provider. Each of these multiplies your cost and, more importantly, your complexity, and complexity causes outages. Teams that adopt multi-region for reliability often find their new failure modes outnumber the ones they removed. Single-region with good backups and an honest recovery time is a defensible choice, and saying so plainly is better engineering than an architecture you cannot operate.\n\nThe question to ask is not \"could this fail\" but \"if this fails for four hours, what does it cost us, and is that more than the standing cost of the alternative\".\n\n## Say something, and say it somewhere they can reach\n\nA status page hosted on the same infrastructure as your product is not a status page. Use a separate provider (several are free at small scale) or a static page on different hosting, or a social account. Whatever it is, it should be linked from your normal site so people know where to look before they need it.\n\nThe message during a third-party outage should say what is affected, what still works, that the cause is upstream, and when you will next update. It should not blame the vendor by way of excuse. Your users chose you, not them; the dependency was your decision.\n\n## Afterwards\n\nWrite it up like any other incident. The useful actions are almost never \"switch provider\" and almost always the smaller ones: add the flag that would have let you degrade, set `stale-if-error`, move the status page off the affected infrastructure, add the dependency to the dashboard you check first. Those are what shorten the next one."},"check":{"question":"Your image-processing provider is down. Your site returns 500 on every page because the header renders user avatars through that provider. What is the most effective immediate action?","options":[{"text":"Turn off the avatar feature behind a flag and serve a placeholder, restoring the rest of the site","correct":true},{"text":"Fail over to a second image provider configured during the incident","correct":false,"tempting":"Adding an integration under pressure is slow and untested; redundancy has to exist before the outage to be usable during it."},{"text":"Increase the timeout on the provider calls so requests have time to succeed","correct":false,"tempting":"Longer timeouts hold workers for longer against a dependency that is down, which makes the collapse worse rather than better."},{"text":"Publish a status update explaining the upstream outage and wait for recovery","correct":false,"tempting":"Communication is necessary but not a fix; a page that could be serving 95% of its function should not stay at zero while you wait."}]},"prev":{"slug":"failing-well","title":"Timeouts, retries, and the failure you made worse","url":"https://addaly.com/learn/cloud-and-shipping/failing-well"},"next":{"slug":"the-first-thirty-minutes","title":"The first thirty minutes of an outage","url":"https://addaly.com/learn/cloud-and-shipping/the-first-thirty-minutes"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The outage that is not yours', in Shipping It. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/cloud-and-shipping/when-your-provider-goes-down (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The outage that is not yours","courseTitle":"Shipping It","module":{"index":5,"slug":"staying-alive","title":"Staying alive: the bad day and the day after"},"answer":null,"keyIdea":"When the broken thing belongs to somebody else, your work is to prove it quickly, degrade or serve stale rather than wait, and communicate from infrastructure that does not share the failure. And most small projects should buy cheap redundancy for stateless providers rather than an architecture they cannot operate.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"sooner-or-later-it-is-not-your-fault","runs":[{"text":"Sooner or later it is not your fault"}]},{"t":"p","runs":[{"text":"At some point the thing that is broken is a certificate authority, a DNS provider, a cloud region, a payment gateway or an identity provider, and nothing in your repository can fix it. This has happened at industry scale several times: a configuration change at a single CDN taking a large share of the web offline for an hour, a DNS provider under attack making dozens of well-known services unreachable, a cloud region failing and taking down the products built on it, including, memorably, some of the status pages meant to report the failure."}]},{"t":"p","runs":[{"text":"Your job during those hours is not to fix somebody else’s system. It is to know quickly that it is theirs, to keep as much of your product working as you can, and to tell your users something true."}]},{"t":"h2","id":"establishing-that-it-is-them-in-five-minutes","runs":[{"text":"Establishing that it is them, in five minutes"}]},{"t":"p","runs":[{"text":"A rough order that usually settles it:"}]},{"t":"ol","start":1,"items":[[{"text":"Check from outside your own network.","b":true},{"text":" A phone on mobile data, or a free external checker. Half of “the site is down” reports are one office’s wifi or one ISP’s DNS resolver."}],[{"text":"Check the dependency directly.","b":true},{"text":" "},{"text":"curl","code":true},{"text":" its health endpoint or an unauthenticated URL from your server. Time it."}],[{"text":"Check their status page and their social feed.","b":true},{"text":" Status pages lag, often by twenty minutes, because a human has to publish. The social feed is usually faster, and third-party outage trackers faster still."}],[{"text":"Check your own dependency dashboard","b":true},{"text":": the second screen from the dashboards lesson exists for precisely this moment."}]]},{"t":"p","runs":[{"text":"Write the answer down with a timestamp as you go. In the postmortem, “we knew it was the provider at 14:07” is a much better sentence than “we think it was maybe an hour”."}]},{"t":"h2","id":"what-you-can-do-ranked-by-how-much-it-actually-helps","runs":[{"text":"What you can do, ranked by how much it actually helps"}]},{"t":"ul","items":[[{"text":"Degrade the feature, keep the site.","b":true},{"text":" The failing-well lesson’s flag per dependency pays for itself here. Turning off search or recommendations is a ten-second action that keeps everything else alive."}],[{"text":"Queue the work.","b":true},{"text":" If the dependency is an email or webhook target, buffer and retry later. Users do not need those to be synchronous, and they rarely notice a twenty-minute delay."}],[{"text":"Serve stale.","b":true},{"text":" A cache configured with "},{"text":"stale-if-error","code":true},{"text":" will keep answering from an old copy while the origin or an upstream is failing. This one line of configuration converts a total outage into slightly out-of-date content."}],[{"text":"Switch provider, if you already can.","b":true},{"text":" A second mail provider or model provider behind one interface is a cheap redundancy, because both are stateless HTTP APIs. This is realistic for mail, SMS, models and image processing."}],[{"text":"Wait, and communicate.","b":true},{"text":" For DNS, identity or your database’s region, there is often no alternative and pretending otherwise wastes the hour."}]]},{"t":"h2","id":"what-is-worth-being-redundant-and-what-is-not","runs":[{"text":"What is worth being redundant, and what is not"}]},{"t":"p","runs":[{"text":"An honest ranking for a small project:"}]},{"t":"p","runs":[{"text":"Worth it, because it is cheap:","b":true},{"text":" a second DNS provider (the protocol supports multiple authoritative servers), a second mail sender, a second model or API provider behind your own interface, backups stored at a different company from the database."}]},{"t":"p","runs":[{"text":"Not worth it for most:","b":true},{"text":" multi-region active-active, multi-cloud, a standby database on another provider. Each of these multiplies your cost and, more importantly, your complexity, and complexity causes outages. Teams that adopt multi-region for reliability often find their new failure modes outnumber the ones they removed. Single-region with good backups and an honest recovery time is a defensible choice, and saying so plainly is better engineering than an architecture you cannot operate."}]},{"t":"p","runs":[{"text":"The question to ask is not “could this fail” but “if this fails for four hours, what does it cost us, and is that more than the standing cost of the alternative”."}]},{"t":"h2","id":"say-something-and-say-it-somewhere-they-can-reach","runs":[{"text":"Say something, and say it somewhere they can reach"}]},{"t":"p","runs":[{"text":"A status page hosted on the same infrastructure as your product is not a status page. Use a separate provider (several are free at small scale) or a static page on different hosting, or a social account. Whatever it is, it should be linked from your normal site so people know where to look before they need it."}]},{"t":"p","runs":[{"text":"The message during a third-party outage should say what is affected, what still works, that the cause is upstream, and when you will next update. It should not blame the vendor by way of excuse. Your users chose you, not them; the dependency was your decision."}]},{"t":"h2","id":"afterwards","runs":[{"text":"Afterwards"}]},{"t":"p","runs":[{"text":"Write it up like any other incident. The useful actions are almost never “switch provider” and almost always the smaller ones: add the flag that would have let you degrade, set "},{"text":"stale-if-error","code":true},{"text":", move the status page off the affected infrastructure, add the dependency to the dashboard you check first. Those are what shorten the next one."}]}],"check":{"question":[{"text":"Your image-processing provider is down. Your site returns 500 on every page because the header renders user avatars through that provider. What is the most effective immediate action?"}],"options":[{"text":[{"text":"Turn off the avatar feature behind a flag and serve a placeholder, restoring the rest of the site"}],"correct":true,"tempting":null},{"text":[{"text":"Fail over to a second image provider configured during the incident"}],"correct":false,"tempting":[{"text":"Adding an integration under pressure is slow and untested; redundancy has to exist before the outage to be usable during it."}]},{"text":[{"text":"Increase the timeout on the provider calls so requests have time to succeed"}],"correct":false,"tempting":[{"text":"Longer timeouts hold workers for longer against a dependency that is down, which makes the collapse worse rather than better."}]},{"text":[{"text":"Publish a status update explaining the upstream outage and wait for recovery"}],"correct":false,"tempting":[{"text":"Communication is necessary but not a fix; a page that could be serving 95% of its function should not stay at zero while you wait."}]}],"order":[2,3,1,0],"why":null},"prev":{"slug":"failing-well","title":"Timeouts, retries, and the failure you made worse","minutes":9},"next":{"slug":"the-first-thirty-minutes","title":"The first thirty minutes of an outage","minutes":10}}}