{"course":{"slug":"building-apps-with-ai","title":"Building Apps with AI","url":"https://addaly.com/learn/building-apps-with-ai","level":"beginner","track":"build"},"lesson":{"slug":"when-the-model-is-down-or-slow","title":"When the model is down, slow or empty","url":"https://addaly.com/learn/building-apps-with-ai/when-the-model-is-down-or-slow","position":64,"of":87,"minutes":7,"answer":"The provider will have a bad hour, so the feature degrades on its own (a bounded retry, a second model behind the same interface.","keyIdea":"The provider will have a bad hour, so the feature degrades on its own (a bounded retry, a second model behind the same interface, or an honest unavailable message) and the rest of the app carries on."},"body":{"format":"markdown","text":"## Providers go down\n\nEvery model provider has outages, rate-limit storms and slow hours, and the status pages record several a month. A feature built as if the call always succeeds turns each of those into an outage of your app. The aim of this lesson is smaller than reliability: it is that the AI feature fails on its own and nothing else notices.\n\n## What \"failed\" looks like\n\nFour shapes, and they need different handling.\n\n**A 429.** Too many requests. On a free tier this is routine and means wait; on a paid tier it means a burst or an exhausted balance. Retryable, after a pause.\n\n**A 5xx or a network error.** The provider is unwell. Retryable, once or twice, with growing waits.\n\n**A timeout.** The call is taking longer than your ceiling. Abort it (the previous lesson set the timer) and treat as a failure. Do not retry a timeout more than once; a slow provider retried three times is a request that takes two minutes and then fails.\n\n**A 200 with nothing in it.** Empty content, or content that fails validation twice. Not retryable in the same way; it is the model declining or producing garbage for this input, and the answer is the fallback below.\n\n## The bounded retry\n\n```typescript\nasync function withRetry<T>(fn: () => Promise<T>, attempts = 3): Promise<T> {\n  for (let i = 0; ; i++) {\n    try { return await fn(); }\n    catch (e) {\n      if (i >= attempts - 1 || !isRetryable(e)) throw e;\n      await sleep(500 * 2 ** i + Math.random() * 200);   // 0.5s, 1s, 2s, with jitter\n    }\n  }\n}\n```\n\nThree attempts, delays that double, a little randomness so a thousand users do not all retry in the same millisecond, and only for errors that are worth retrying. The generated version is often a `while` loop with no counter and no delay, and the question above is what that does during an outage: each request holds a serverless function for its whole timeout, then holds another, and the host's concurrency fills with functions waiting on a provider that is not answering. Pages that never touch the model become slow because there is no capacity left to serve them.\n\n## A second model behind the same door\n\nBecause most providers speak the same request format, a fallback is a second base URL and key. When the primary fails after its retries, try the secondary once:\n\n```typescript\nconst providers = [\n  { url: process.env.LLM_BASE_URL, key: process.env.LLM_API_KEY, model: process.env.LLM_MODEL },\n  { url: process.env.FALLBACK_BASE_URL, key: process.env.FALLBACK_API_KEY, model: process.env.FALLBACK_MODEL },\n];\n```\n\nThe secondary can be a cheaper model, a different company, or a router service such as OpenRouter that does the failover itself. The quality may drop for the duration. That is a better hour than no feature, and the log line from the prompt lesson records which model answered, so you can see afterwards how often the fallback carried the load.\n\n## When there is no answer\n\nAfter retries and fallback, sometimes there is nothing. Say so:\n\n> The summary is not available right now. Your note is saved; try again in a few minutes.\n\nThree properties of that sentence. It is honest. It tells the user what did happen (the note is saved), so they do not retype it. And it does not pretend. The version that shows a generic phrase as if the model produced it, or shows an empty box, teaches users that the feature is unreliable in a way they cannot name.\n\n> Figure: One click during a provider's bad hour. Timeline: 0 s, First attempt. A 429 or a 5xx is retryable; a 400 is not, and is thrown at once.; 0.5 s, Second attempt, after a short wait with jitter added.; 1.5 s, Third and last attempt on the primary provider.; 3.5 s, The fallback provider, once, behind the same interface. The log records which model answered.; after that, The summary is not available right now. Your note is saved; try again in a few minutes.. Three attempts, delays that double, a little randomness so a thousand users do not retry in the same millisecond, and then an honest sentence. The generated version is a while loop with no counter.\n\nA cached previous result is a reasonable fallback for features where one exists: yesterday's category suggestion, the last summary of this document. Label it as such.\n\n## Degrade the feature, not the app\n\nThe note saves before the summary is requested, not after. The page renders with the summary slot empty and fills it when the call returns. A booking is confirmed by the database, and the model's suggested reply is a decoration on top. If the model vanished from the internet tomorrow, every core action in the app should still complete, and the only visible change should be some empty boxes with honest sentences in them.\n\nCheck that by breaking it on purpose: set the base URL to something that does not resolve, and click through the app. Anything that stops working was built with the model on the critical path, and moving it off is a design change worth making before the first real outage does it for you."},"check":{"question":"A route retries a model call on any error. During a provider outage, the host bill spikes and the app's other pages become slow. What is the mechanism?","options":[{"text":"The provider charges for failed requests, and the retries multiplied the number of failures.","correct":false,"tempting":"Most providers do not bill failed calls; the cost here is on your side, in function time, not on the provider's invoice."},{"text":"The retries were sent from the browser and consumed the users' bandwidth.","correct":false,"tempting":"The calls are server-side; the browser is waiting on one request, not sending many."},{"text":"The outage caused the database to slow down, because every failed request is being written to the log table with its full request body, and the table has grown large enough that it now needs a bigger plan.","correct":false,"tempting":"Logging a few hundred failures is trivial for a database; the slowdown is in compute capacity, not storage."},{"text":"Each stuck request held a function open for its full timeout, retried, and held it again, so the host's concurrent capacity filled with waiting AI calls and every other request queued behind them.","correct":true}]},"prev":{"slug":"capping-what-one-user-can-cost","title":"Capping what one user can cost you","url":"https://addaly.com/learn/building-apps-with-ai/capping-what-one-user-can-cost"},"next":{"slug":"giving-the-model-your-own-data","title":"Giving the model your own data","url":"https://addaly.com/learn/building-apps-with-ai/giving-the-model-your-own-data"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'When the model is down, slow or empty', in Building Apps with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-apps-with-ai/when-the-model-is-down-or-slow (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"When the model is down, slow or empty","courseTitle":"Building Apps with AI","module":{"index":7,"slug":"a-model-inside-your-app","title":"Putting a model inside your own app"},"answer":null,"keyIdea":"The provider will have a bad hour, so the feature degrades on its own (a bounded retry, a second model behind the same interface, or an honest unavailable message) and the rest of the app carries on.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"timeline","title":"One click during a provider’s bad hour","events":[{"when":"0 s","what":"First attempt. A 429 or a 5xx is retryable; a 400 is not, and is thrown at once."},{"when":"0.5 s","what":"Second attempt, after a short wait with jitter added."},{"when":"1.5 s","what":"Third and last attempt on the primary provider."},{"when":"3.5 s","what":"The fallback provider, once, behind the same interface. The log records which model answered."},{"when":"after that","what":"The summary is not available right now. Your note is saved; try again in a few minutes."}],"caption":"Three attempts, delays that double, a little randomness so a thousand users do not retry in the same millisecond, and then an honest sentence. The generated version is a while loop with no counter."},"alt":"Figure: One click during a provider’s bad hour. Timeline: 0 s, First attempt. A 429 or a 5xx is retryable; a 400 is not, and is thrown at once.; 0.5 s, Second attempt, after a short wait with jitter added.; 1.5 s, Third and last attempt on the primary provider.; 3.5 s, The fallback provider, once, behind the same interface. The log records which model answered.; after that, The summary is not available right now. Your note is saved; try again in a few minutes.. Three attempts, delays that double, a little randomness so a thousand users do not retry in the same millisecond, and then an honest sentence. The generated version is a while loop with no counter."}},"blocks":[{"t":"h2","id":"providers-go-down","runs":[{"text":"Providers go down"}]},{"t":"p","runs":[{"text":"Every model provider has outages, rate-limit storms and slow hours, and the status pages record several a month. A feature built as if the call always succeeds turns each of those into an outage of your app. The aim of this lesson is smaller than reliability: it is that the AI feature fails on its own and nothing else notices."}]},{"t":"h2","id":"what-failed-looks-like","runs":[{"text":"What “failed” looks like"}]},{"t":"p","runs":[{"text":"Four shapes, and they need different handling."}]},{"t":"p","runs":[{"text":"A 429.","b":true},{"text":" Too many requests. On a free tier this is routine and means wait; on a paid tier it means a burst or an exhausted balance. Retryable, after a pause."}]},{"t":"p","runs":[{"text":"A 5xx or a network error.","b":true},{"text":" The provider is unwell. Retryable, once or twice, with growing waits."}]},{"t":"p","runs":[{"text":"A timeout.","b":true},{"text":" The call is taking longer than your ceiling. Abort it (the previous lesson set the timer) and treat as a failure. Do not retry a timeout more than once; a slow provider retried three times is a request that takes two minutes and then fails."}]},{"t":"p","runs":[{"text":"A 200 with nothing in it.","b":true},{"text":" Empty content, or content that fails validation twice. Not retryable in the same way; it is the model declining or producing garbage for this input, and the answer is the fallback below."}]},{"t":"h2","id":"the-bounded-retry","runs":[{"text":"The bounded retry"}]},{"t":"code","lang":"typescript","label":"TypeScript","text":"async function withRetry<T>(fn: () => Promise<T>, attempts = 3): Promise<T> {\n  for (let i = 0; ; i++) {\n    try { return await fn(); }\n    catch (e) {\n      if (i >= attempts - 1 || !isRetryable(e)) throw e;\n      await sleep(500 * 2 ** i + Math.random() * 200);   // 0.5s, 1s, 2s, with jitter\n    }\n  }\n}"},{"t":"p","runs":[{"text":"Three attempts, delays that double, a little randomness so a thousand users do not all retry in the same millisecond, and only for errors that are worth retrying. The generated version is often a "},{"text":"while","code":true},{"text":" loop with no counter and no delay, and the question above is what that does during an outage: each request holds a serverless function for its whole timeout, then holds another, and the host’s concurrency fills with functions waiting on a provider that is not answering. Pages that never touch the model become slow because there is no capacity left to serve them."}]},{"t":"h2","id":"a-second-model-behind-the-same-door","runs":[{"text":"A second model behind the same door"}]},{"t":"p","runs":[{"text":"Because most providers speak the same request format, a fallback is a second base URL and key. When the primary fails after its retries, try the secondary once:"}]},{"t":"code","lang":"typescript","label":"TypeScript","text":"const providers = [\n  { url: process.env.LLM_BASE_URL, key: process.env.LLM_API_KEY, model: process.env.LLM_MODEL },\n  { url: process.env.FALLBACK_BASE_URL, key: process.env.FALLBACK_API_KEY, model: process.env.FALLBACK_MODEL },\n];"},{"t":"p","runs":[{"text":"The secondary can be a cheaper model, a different company, or a router service such as OpenRouter that does the failover itself. The quality may drop for the duration. That is a better hour than no feature, and the log line from the prompt lesson records which model answered, so you can see afterwards how often the fallback carried the load."}]},{"t":"h2","id":"when-there-is-no-answer","runs":[{"text":"When there is no answer"}]},{"t":"p","runs":[{"text":"After retries and fallback, sometimes there is nothing. Say so:"}]},{"t":"quote","runs":[{"text":"The summary is not available right now. Your note is saved; try again in a few minutes."}]},{"t":"p","runs":[{"text":"Three properties of that sentence. It is honest. It tells the user what did happen (the note is saved), so they do not retype it. And it does not pretend. The version that shows a generic phrase as if the model produced it, or shows an empty box, teaches users that the feature is unreliable in a way they cannot name."}]},{"t":"p","runs":[{"text":"A cached previous result is a reasonable fallback for features where one exists: yesterday’s category suggestion, the last summary of this document. Label it as such."}]},{"t":"h2","id":"degrade-the-feature-not-the-app","runs":[{"text":"Degrade the feature, not the app"}]},{"t":"p","runs":[{"text":"The note saves before the summary is requested, not after. The page renders with the summary slot empty and fills it when the call returns. A booking is confirmed by the database, and the model’s suggested reply is a decoration on top. If the model vanished from the internet tomorrow, every core action in the app should still complete, and the only visible change should be some empty boxes with honest sentences in them."}]},{"t":"p","runs":[{"text":"Check that by breaking it on purpose: set the base URL to something that does not resolve, and click through the app. Anything that stops working was built with the model on the critical path, and moving it off is a design change worth making before the first real outage does it for you."}]}],"check":{"question":[{"text":"A route retries a model call on any error. During a provider outage, the host bill spikes and the app’s other pages become slow. What is the mechanism?"}],"options":[{"text":[{"text":"The provider charges for failed requests, and the retries multiplied the number of failures."}],"correct":false,"tempting":[{"text":"Most providers do not bill failed calls; the cost here is on your side, in function time, not on the provider’s invoice."}]},{"text":[{"text":"The retries were sent from the browser and consumed the users’ bandwidth."}],"correct":false,"tempting":[{"text":"The calls are server-side; the browser is waiting on one request, not sending many."}]},{"text":[{"text":"The outage caused the database to slow down, because every failed request is being written to the log table with its full request body, and the table has grown large enough that it now needs a bigger plan."}],"correct":false,"tempting":[{"text":"Logging a few hundred failures is trivial for a database; the slowdown is in compute capacity, not storage."}]},{"text":[{"text":"Each stuck request held a function open for its full timeout, retried, and held it again, so the host’s concurrent capacity filled with waiting AI calls and every other request queued behind them."}],"correct":true,"tempting":null}],"order":[0,3,2,1],"why":null},"prev":{"slug":"capping-what-one-user-can-cost","title":"Capping what one user can cost you","minutes":8},"next":{"slug":"giving-the-model-your-own-data","title":"Giving the model your own data","minutes":10}}}