{"course":{"slug":"cloud-and-shipping","title":"Shipping It","url":"https://addaly.com/learn/cloud-and-shipping","level":"intermediate","track":"build"},"lesson":{"slug":"the-fallback-when-the-model-is-down","title":"What the page says when the model is not there","url":"https://addaly.com/learn/cloud-and-shipping/the-fallback-when-the-model-is-down","position":57,"of":74,"minutes":9,"answer":"Model calls fail slowly, partially and expensively, so set idle timeouts on streams, validate the shape of every answer.","keyIdea":"Model calls fail slowly, partially and expensively, so set idle timeouts on streams, validate the shape of every answer, define one cheaper fallback per feature and write down in advance whether each gate fails open or closed."},"body":{"format":"markdown","text":"## Model dependencies fail differently\n\nEverything the failing-well lesson said about timeouts and breakers applies here, with four differences that change the settings.\n\n1. **Normal latency is seconds, not milliseconds.** A 2-second timeout is aggressive for a database and far too tight for generation. Timeouts must be set per operation: a classification call might be 3 seconds, a long generation 60.\n2. **Streaming complicates the timeout.** A stream that has produced tokens is alive even after 30 seconds. What you want is an idle timeout (no bytes for 10 seconds) rather than a total deadline, plus a maximum total as a backstop.\n3. **Failure is often partial.** The call succeeded and returned something unusable: truncated output, invalid JSON, a refusal, an empty string. HTTP 200 is not success for a model. Validate the shape of the answer before you use it, and count schema failures as errors in your metrics.\n4. **Retries are expensive.** Retrying a database query costs a millisecond; retrying a generation costs cents and seconds. Retry once on a clear transient signal, and prefer falling back to something cheaper over trying the same expensive thing again.\n\n## Decide the degraded answer before you need it\n\nFor every model-backed feature, write down what happens when the model is unavailable. The options, roughly in order of how well they preserve the product:\n\n- **Serve a cached answer.** Exact-match cache from the cost lesson doubles as an outage buffer, and `stale-if-error` semantics apply to your own cache too.\n- **Fall back to a cheaper or local model.** A small model on your own hardware, or a smaller model from the same provider, is worse and available. For classification this is often nearly as good.\n- **Fall back to rules.** A keyword filter, a heuristic ranking, most-recent order. Unfashionable, deterministic, and it keeps the feature alive.\n- **Fall back to a second provider.** Model APIs are mostly interchangeable behind one interface, and this is one of the cheapest redundancies available anywhere in this course. Keep the prompt and the parsing shared, and test the second path monthly: an untested fallback is not a fallback.\n- **Queue it.** If the answer is not needed instantly, accept the request and process later.\n- **Turn the feature off** behind a flag and say so plainly on the page.\n\nWhat should not happen is a 500 on a page whose main content does not need the model. The header, the article, the form: all of those work without it.\n\n## Fail open or fail closed, and why it is not a preference\n\nWhen the model gates something, its unavailability forces a choice.\n\n- A **spam classifier** that is down: failing open lets spam through; failing closed blocks legitimate users. Most products fail open for trusted accounts and closed for new ones, which is a defensible split rather than a single rule.\n- A **safety check** in front of published content: failing open publishes unreviewed material to everybody. That is usually the worse outcome, so the honest answer is to hold the content and tell the author it is pending.\n- A **payment risk score** that is down: failing open takes fraud, failing closed refuses revenue. This one is a business decision and should be made by whoever owns the money, in advance, in writing.\n\nThe pattern worth taking away: the answer depends on which error is more expensive and on who is affected, and it should be decided calmly and written into the runbook, not improvised at midnight by whoever is on call.\n\n## Guardrails cost latency too\n\nIf you check the model's output with another model (a safety classifier, a schema validator, a second opinion) you have added a dependency to the critical path. Two consequences: your latency is now the sum, and your availability is now the product. Run guardrails in parallel with the main call where the design allows, keep them small and fast, and give them their own timeout with a defined behaviour when they time out.\n\n## Test it deliberately\n\nAdd a drill to the list from the practising-failure lesson: block the model provider at the firewall for five minutes during working hours and watch what your product does. The first time, something will be wrong: usually an unset timeout, a fallback that was never wired, or an error page where a degraded page should be. That is exactly the finding you want, on a Tuesday, rather than during the provider's next incident."},"check":{"question":"Your generation endpoint uses a 30-second total timeout. During a provider incident, users see requests hang for the full 30 seconds and then fail, even though the stream stopped producing tokens after 2 seconds. What change addresses this directly?","options":[{"text":"Reduce the total timeout to 5 seconds","correct":false,"tempting":"It shortens the hang but also cuts off legitimate long generations, trading one failure for another."},{"text":"Add an idle timeout that fails when no bytes arrive for a few seconds, keeping the total as a backstop","correct":true},{"text":"Retry the request automatically when it times out","correct":false,"tempting":"Doubles the wait and the cost during an incident, and retrying an unhealthy provider is the retry-storm pattern."},{"text":"Disable streaming so the response either arrives whole or fails immediately","correct":false,"tempting":"Without streaming the request still waits for the full timeout, and users lose the perceived-latency benefit for no gain."}]},"prev":{"slug":"feedback-loops","title":"The model that trains on its own decisions","url":"https://addaly.com/learn/cloud-and-shipping/feedback-loops"},"next":{"slug":"who-is-allowed-to-do-what","title":"Authentication is not authorisation","url":"https://addaly.com/learn/cloud-and-shipping/who-is-allowed-to-do-what"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What the page says when the model is not there', in Shipping It. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/cloud-and-shipping/the-fallback-when-the-model-is-down (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What the page says when the model is not there","courseTitle":"Shipping It","module":{"index":6,"slug":"shipping-a-model","title":"Shipping a model, not just a web app"},"answer":null,"keyIdea":"Model calls fail slowly, partially and expensively, so set idle timeouts on streams, validate the shape of every answer, define one cheaper fallback per feature and write down in advance whether each gate fails open or closed.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"model-dependencies-fail-differently","runs":[{"text":"Model dependencies fail differently"}]},{"t":"p","runs":[{"text":"Everything the failing-well lesson said about timeouts and breakers applies here, with four differences that change the settings."}]},{"t":"ol","start":1,"items":[[{"text":"Normal latency is seconds, not milliseconds.","b":true},{"text":" A 2-second timeout is aggressive for a database and far too tight for generation. Timeouts must be set per operation: a classification call might be 3 seconds, a long generation 60."}],[{"text":"Streaming complicates the timeout.","b":true},{"text":" A stream that has produced tokens is alive even after 30 seconds. What you want is an idle timeout (no bytes for 10 seconds) rather than a total deadline, plus a maximum total as a backstop."}],[{"text":"Failure is often partial.","b":true},{"text":" The call succeeded and returned something unusable: truncated output, invalid JSON, a refusal, an empty string. HTTP 200 is not success for a model. Validate the shape of the answer before you use it, and count schema failures as errors in your metrics."}],[{"text":"Retries are expensive.","b":true},{"text":" Retrying a database query costs a millisecond; retrying a generation costs cents and seconds. Retry once on a clear transient signal, and prefer falling back to something cheaper over trying the same expensive thing again."}]]},{"t":"h2","id":"decide-the-degraded-answer-before-you-need-it","runs":[{"text":"Decide the degraded answer before you need it"}]},{"t":"p","runs":[{"text":"For every model-backed feature, write down what happens when the model is unavailable. The options, roughly in order of how well they preserve the product:"}]},{"t":"ul","items":[[{"text":"Serve a cached answer.","b":true},{"text":" Exact-match cache from the cost lesson doubles as an outage buffer, and "},{"text":"stale-if-error","code":true},{"text":" semantics apply to your own cache too."}],[{"text":"Fall back to a cheaper or local model.","b":true},{"text":" A small model on your own hardware, or a smaller model from the same provider, is worse and available. For classification this is often nearly as good."}],[{"text":"Fall back to rules.","b":true},{"text":" A keyword filter, a heuristic ranking, most-recent order. Unfashionable, deterministic, and it keeps the feature alive."}],[{"text":"Fall back to a second provider.","b":true},{"text":" Model APIs are mostly interchangeable behind one interface, and this is one of the cheapest redundancies available anywhere in this course. Keep the prompt and the parsing shared, and test the second path monthly: an untested fallback is not a fallback."}],[{"text":"Queue it.","b":true},{"text":" If the answer is not needed instantly, accept the request and process later."}],[{"text":"Turn the feature off","b":true},{"text":" behind a flag and say so plainly on the page."}]]},{"t":"p","runs":[{"text":"What should not happen is a 500 on a page whose main content does not need the model. The header, the article, the form: all of those work without it."}]},{"t":"h2","id":"fail-open-or-fail-closed-and-why-it-is-not-a-preference","runs":[{"text":"Fail open or fail closed, and why it is not a preference"}]},{"t":"p","runs":[{"text":"When the model gates something, its unavailability forces a choice."}]},{"t":"ul","items":[[{"text":"A "},{"text":"spam classifier","b":true},{"text":" that is down: failing open lets spam through; failing closed blocks legitimate users. Most products fail open for trusted accounts and closed for new ones, which is a defensible split rather than a single rule."}],[{"text":"A "},{"text":"safety check","b":true},{"text":" in front of published content: failing open publishes unreviewed material to everybody. That is usually the worse outcome, so the honest answer is to hold the content and tell the author it is pending."}],[{"text":"A "},{"text":"payment risk score","b":true},{"text":" that is down: failing open takes fraud, failing closed refuses revenue. This one is a business decision and should be made by whoever owns the money, in advance, in writing."}]]},{"t":"p","runs":[{"text":"The pattern worth taking away: the answer depends on which error is more expensive and on who is affected, and it should be decided calmly and written into the runbook, not improvised at midnight by whoever is on call."}]},{"t":"h2","id":"guardrails-cost-latency-too","runs":[{"text":"Guardrails cost latency too"}]},{"t":"p","runs":[{"text":"If you check the model’s output with another model (a safety classifier, a schema validator, a second opinion) you have added a dependency to the critical path. Two consequences: your latency is now the sum, and your availability is now the product. Run guardrails in parallel with the main call where the design allows, keep them small and fast, and give them their own timeout with a defined behaviour when they time out."}]},{"t":"h2","id":"test-it-deliberately","runs":[{"text":"Test it deliberately"}]},{"t":"p","runs":[{"text":"Add a drill to the list from the practising-failure lesson: block the model provider at the firewall for five minutes during working hours and watch what your product does. The first time, something will be wrong: usually an unset timeout, a fallback that was never wired, or an error page where a degraded page should be. That is exactly the finding you want, on a Tuesday, rather than during the provider’s next incident."}]}],"check":{"question":[{"text":"Your generation endpoint uses a 30-second total timeout. During a provider incident, users see requests hang for the full 30 seconds and then fail, even though the stream stopped producing tokens after 2 seconds. What change addresses this directly?"}],"options":[{"text":[{"text":"Reduce the total timeout to 5 seconds"}],"correct":false,"tempting":[{"text":"It shortens the hang but also cuts off legitimate long generations, trading one failure for another."}]},{"text":[{"text":"Add an idle timeout that fails when no bytes arrive for a few seconds, keeping the total as a backstop"}],"correct":true,"tempting":null},{"text":[{"text":"Retry the request automatically when it times out"}],"correct":false,"tempting":[{"text":"Doubles the wait and the cost during an incident, and retrying an unhealthy provider is the retry-storm pattern."}]},{"text":[{"text":"Disable streaming so the response either arrives whole or fails immediately"}],"correct":false,"tempting":[{"text":"Without streaming the request still waits for the full timeout, and users lose the perceived-latency benefit for no gain."}]}],"order":[2,1,0,3],"why":null},"prev":{"slug":"feedback-loops","title":"The model that trains on its own decisions","minutes":9},"next":{"slug":"who-is-allowed-to-do-what","title":"Authentication is not authorisation","minutes":9}}}