Routing and cascades: paying the high price only where it is needed

The shape of real traffic

Look at a week of requests to any assistant and most of them are simple: a factual question with a clear answer in the corpus, a greeting, a request to rephrase something. A small fraction are hard: multi-step reasoning, an ambiguous question, a long document, a tool sequence. The frontier model earns its price on the hard fraction and wastes it on the rest. Routing is sending each request to the cheapest model that handles it, and it is the largest cost lever after the batch window in most products.

Two designs

A router decides up front. A classifier looks at the request and picks the model before any generation. The classifier can be heuristics (input length, presence of an attachment, a tool-requiring intent, a language the small model handles poorly) or a tiny model asked to label the request as easy or hard, or the intent label you already compute for analytics. It costs a few milliseconds and a fraction of a cent.

A cascade escalates on failure. The small model answers first. If the answer fails a check (the structured output did not validate, the model refused, the citation check failed, a cheap judge scored it low, the reranker found nothing) the request is re-run on the large model. The small model’s attempt is wasted on escalated requests, so the cascade pays extra on the hard fraction and saves on the easy one.

The router is better when hardness is predictable from the input; the cascade is better when it is only visible in the output. Many systems use both: route the obvious cases, cascade the rest.

The arithmetic

Suppose the large model costs ten times the small one per token, and seventy per cent of requests can be handled by the small model. Sending everything to the large model costs 100 units. Routing: 70 requests at 1 unit and 30 at 10 units is 370 units, against 1,000 for all-large: a 63 per cent saving. A cascade that escalates 30 per cent pays 100 small attempts plus 300 large: 400 units, a 60 per cent saving. If the small model handles ninety per cent, the saving approaches ninety per cent. The number that decides it is the easy fraction, and only your evaluation set, run on both models, tells you what it is.

The two failure modes

The escalation never fires. A cascade whose check is “did the small model return something?” escalates nothing, because the small model always returns something. Wrong answers ship at the low price. The check must be able to fail on a wrong answer (validation, citation, a judge) and the rate at which it fires should be reported daily. A rate near zero is a rule that is not working.

The escalation always fires. A check that is too strict sends everything to the large model after paying for a small attempt first, and the cascade costs more than no cascade at all. Watch the escalation rate; if it climbs past the easy fraction the evaluation set predicted, the check or the small model has changed.

Evaluate per route

The harness must know which route each case took. Report pass rates for the small-model slice and the large-model slice separately, and for the escalated cases before and after escalation. An overall pass rate hides a small model that is failing quietly on a slice the router marks easy. Tag the evaluation cases by expected route, and check that the router’s decisions match the tags: a router that sends a case tagged hard to the small model is a finding.

Behaviour must match across routes

Users notice when the assistant changes voice between questions. The two models are given the same system prompt and the same examples, and the evaluation set includes checks on tone and format that both must pass. Where the small model cannot match (a formatting quirk, a weaker grasp of one language) either fix it with a few-shot example for that model or route that slice to the large one.

Where the small model wins outright

Sometimes the small model, with good retrieval, beats the large model on the evaluation set for the easy slice, on speed and on quality both, because the task is a lookup and the large model’s extra capacity is spent on elaboration nobody asked for. Do not assume the expensive model is better for every slice; run the set and read the numbers per slice.

Keep it changeable

Model names, thresholds and the routing rule live in configuration behind the same flags as prompts, from the shipping lesson. The models will change, the price ratio between them will change, and the easy fraction will change as the product does. A routing rule from six months ago, hard-coded, is spending money on a ratio that no longer exists.