Cascades and routers: paying for the hard cases only

The shape of a workload

Look at a day of real requests and most of them are easy: a reformat, a short classification, a question the small model answers as well as the large one. A few are hard. Sending everything to the frontier model pays the frontier price on every easy request in order to be ready for the hard ones, and the previous lessons’ arithmetic says that price is a multiple of ten or more.

Published routing studies, on mixed workloads, report that sending a third to two-thirds of requests to a cheap model kept around 95 per cent of the strong model’s measured quality at somewhere between a third and a half of the cost. The exact figures depend on the workload; the shape does not. There are two ways to build this, and each has a weak point you can name in advance.

Cascade: try cheap, escalate on doubt

The cheap model answers first. Something then decides whether to accept the answer or send the request on to the expensive model. Cost per request is the cheap price plus the escalation rate times the expensive price; latency in the worst case is both calls in sequence.

The whole design rests on the deciding signal, and module six told you what to expect from it. A small model’s verbalised confidence clusters high whatever the answer. Its log-probabilities rank but do not calibrate. Semantic entropy works and costs several samples. So the order of preference is:

1. A deterministic check, when one exists. The JSON validates. The code passes the tests. The extracted date parses. The answer is one of the allowed labels. These are free and exact, and a task that admits one should be structured to admit one: module three’s constrained decoding is often what makes it possible.
2. A calibrated score. The uncertainty lesson’s procedure, applied to the cheap model on labelled traffic: bucket its scores, measure the accuracy per bucket, and set the escalation threshold where the accuracy drops below what you will tolerate.
3. Consistency. Sample the cheap model three times; disagreement escalates. Costs three cheap calls, which is often still far below one expensive one.

The classic failure is a threshold set by feel on an uncalibrated signal. A cascade that escalates four per cent of requests on “confidence under 70” is almost always a cascade whose small model is rarely under 70 about anything, including its mistakes.

Router: decide first, call once

The alternative sends each request to one model, chosen before any answer exists. The chooser is a small classifier (a fine-tuned encoder of the kind module one described, or a nearest-neighbour lookup over embeddings of past requests) trained on your own traffic: for each labelled item, which model got it right at the lowest cost.

A router has no worst-case double latency and no dependence on the cheap model’s self-assessment. It has its own weak points. It needs a few thousand labelled examples from your real distribution. Its accuracy decays as the traffic changes (module six’s input drift) so it is retrained on a schedule. And its errors are silent: a hard request routed cheap gets a confident wrong answer, with nothing downstream to catch it.

Routing without machine learning

Before either of the above, try rules. Route by language (the Hindi lesson’s point about which models handle which languages) by input length, by the presence of code, by the customer tier, by whether the request touches a tool. In many workloads a handful of rules captures most of the saving, is auditable, and never drifts in a way you cannot see in the code. Add the learned router only for the residual that rules cannot separate.

Composing with the rest of the module

Caching sits before routing: a cached prefix is cheap at either model. The batch endpoint is a lane of its own for anything that can wait, whatever its difficulty. Constrained decoding on the cheap model makes its output checkable, which turns a cascade’s weakest link into its strongest. And the cheap model can be a local one from this module’s fourth lesson, at zero marginal cost, which changes the arithmetic of what counts as easy enough.

One caution from the prompt-sensitivity lesson: two models rarely want the same prompt. A routed system has at least two prompts to maintain and evaluate, and a change to one is not a change to the other.

Measuring a routed system

Measure the system, not the models. A reported “95 per cent of quality retained” is an average that hides which five per cent was lost, and it is usually the hardest, most valuable requests. Keep a random sample of requests that were routed cheap, re-run them through the expensive model, and compare, continuously, not once. That sample is the only instrument that tells you whether the router is still doing its job after the traffic moved.

The honest state

Fully automatic routers are still fragile. The version that holds up in production is a few deterministic rules, one calibrated uncertainty gate, a cheap deterministic check wherever the task allows one, and the continuous sample above. Providers have begun routing inside their own products (a request to one model name may be served by several) which is one more mechanism behind module six’s “same name, different behaviour”, and one more reason to keep the canary set running.