When the model is down, slow or empty

Providers go down

Every model provider has outages, rate-limit storms and slow hours, and the status pages record several a month. A feature built as if the call always succeeds turns each of those into an outage of your app. The aim of this lesson is smaller than reliability: it is that the AI feature fails on its own and nothing else notices.

What “failed” looks like

Four shapes, and they need different handling.

A 429. Too many requests. On a free tier this is routine and means wait; on a paid tier it means a burst or an exhausted balance. Retryable, after a pause.

A 5xx or a network error. The provider is unwell. Retryable, once or twice, with growing waits.

A timeout. The call is taking longer than your ceiling. Abort it (the previous lesson set the timer) and treat as a failure. Do not retry a timeout more than once; a slow provider retried three times is a request that takes two minutes and then fails.

A 200 with nothing in it. Empty content, or content that fails validation twice. Not retryable in the same way; it is the model declining or producing garbage for this input, and the answer is the fallback below.

The bounded retry

async function withRetry<T>(fn: () => Promise<T>, attempts = 3): Promise<T> {
  for (let i = 0; ; i++) {
    try { return await fn(); }
    catch (e) {
      if (i >= attempts - 1 || !isRetryable(e)) throw e;
      await sleep(500 * 2 ** i + Math.random() * 200);   // 0.5s, 1s, 2s, with jitter
    }
  }
}

Three attempts, delays that double, a little randomness so a thousand users do not all retry in the same millisecond, and only for errors that are worth retrying. The generated version is often a while loop with no counter and no delay, and the question above is what that does during an outage: each request holds a serverless function for its whole timeout, then holds another, and the host’s concurrency fills with functions waiting on a provider that is not answering. Pages that never touch the model become slow because there is no capacity left to serve them.

A second model behind the same door

Because most providers speak the same request format, a fallback is a second base URL and key. When the primary fails after its retries, try the secondary once:

const providers = [
  { url: process.env.LLM_BASE_URL, key: process.env.LLM_API_KEY, model: process.env.LLM_MODEL },
  { url: process.env.FALLBACK_BASE_URL, key: process.env.FALLBACK_API_KEY, model: process.env.FALLBACK_MODEL },
];

The secondary can be a cheaper model, a different company, or a router service such as OpenRouter that does the failover itself. The quality may drop for the duration. That is a better hour than no feature, and the log line from the prompt lesson records which model answered, so you can see afterwards how often the fallback carried the load.

When there is no answer

After retries and fallback, sometimes there is nothing. Say so:

The summary is not available right now. Your note is saved; try again in a few minutes.

Three properties of that sentence. It is honest. It tells the user what did happen (the note is saved), so they do not retype it. And it does not pretend. The version that shows a generic phrase as if the model produced it, or shows an empty box, teaches users that the feature is unreliable in a way they cannot name.

A cached previous result is a reasonable fallback for features where one exists: yesterday’s category suggestion, the last summary of this document. Label it as such.

Degrade the feature, not the app

The note saves before the summary is requested, not after. The page renders with the summary slot empty and fills it when the call returns. A booking is confirmed by the database, and the model’s suggested reply is a decoration on top. If the model vanished from the internet tomorrow, every core action in the app should still complete, and the only visible change should be some empty boxes with honest sentences in them.

Check that by breaking it on purpose: set the base URL to something that does not resolve, and click through the app. Anything that stops working was built with the model on the critical path, and moving it off is a design change worth making before the first real outage does it for you.