Outages, fallbacks and what does not port

Providers go down

Every provider has outages, and their status pages lag the reality by minutes to hours. The number that matters is yours: the error rate and the latency from your own traces, alerted on your own thresholds. By the time the status page turns amber, your users have been staring at a spinner for a while.

A fallback chain

Primary provider; a secondary on a different provider, not a different model at the same one, because outages are usually provider-wide; then a degraded mode. This site’s own assistant runs through a cross-vendor chain (DeepSeek, then GLM, then Qwen, then Kimi) via a single routing service, so that one company’s bad afternoon is not the product’s.

Decide the fallback by error class. A 5xx or a timeout: fall back now. A 429: back off first, then fall back if it persists. A 400: do not fall back, because the request is malformed and the secondary will reject it too; that is your bug, and a fallback would hide it.

The abstraction layer, and what it cannot abstract

LiteLLM, free and open source, presents many providers behind one request shape; OpenRouter does the same as a hosted service; the Vercel AI SDK does it in the application. They normalise the obvious differences (field names, authentication, streaming formats) and they are worth using. They cannot normalise the rest, and the rest is where fallbacks fail silently.

• Tool calls. The schema dialects differ in detail; support for parallel calls differs; the way arguments are returned differs. A tool definition tuned for one provider can be ignored or mangled by another.
• Structured output. Some providers enforce a schema strictly; others treat it as a hint. A pipeline that relies on strict mode receives near-JSON from the fallback and breaks in the parser.
• Prompt behaviour. The same system prompt produces different formats, different verbosity and different refusal thresholds on different models. A prompt that never refuses on the primary may refuse a tenth of requests on the secondary.
• Tokenisers. Different models count tokens differently, so the context budget, the trimming logic and the cost model all shift. A conversation that fits on the primary can exceed the window on the secondary.
• Stop reasons and usage fields. Names and semantics differ; code that checks a stop reason by string may never match on the fallback.
• Caching. Prefix caching works differently or not at all, so cost and first-token latency jump.

The harness is the portability test

Run the evaluation suite against the fallback model on a schedule (monthly is enough) and keep its pass rate beside the primary’s. Where the fallback diverges, fix it with model-specific overrides in configuration: a different few-shot example, a different output instruction, a stricter parser. Never a forked prompt; one prompt with per-model overrides, in version control, is maintainable, and two prompts drift apart within a quarter.

Break it on purpose

A fallback that has never fired will not fire when it is needed. In staging, weekly, revoke the primary key or point it at a black hole, and watch the chain engage. Confirm from traces that the secondary served the requests, that structured output still parsed, that tool calls still worked, and that the interface showed nothing strange. The first time this test is run it almost always finds something; that is the point of running it.

Degrade honestly

When the whole chain is down, the options in order of preference: serve an exact-cache hit if there is one; tell the user plainly that the assistant is unavailable and offer the non-AI path: search, a form, a person; and never let a request quietly fall through to a model that does not understand the tool schema and returns a fluent, wrong answer. A visible outage costs a little trust; a silent wrong answer costs more, and it is not reported for days. Show the degraded state in the interface, as a banner, while it lasts.

Falling back sends data elsewhere

A fallback provider is another company receiving your users’ messages and documents. The lesson on what you send to the provider applies to every name in the chain: the privacy notice must name all of them, each needs the same data-processing terms, and a fallback that moves data out of a region you promised to keep it in is not a fallback but a breach. If residency is a commitment, the secondary must be in the same region, and if no such secondary exists, the honest degraded mode is the only one available.

Regional versus global

Some outages take out one provider region and not others; a second region at the same provider is a cheaper secondary than a second provider, where residency allows it, and it shares the tool schema and the tokeniser, which removes most of the porting problems above. Use it as the first hop, and a different provider as the second.