Not every request needs the big model

The distribution is not flat

Look at a day of traffic and you will usually find the same shape: a large majority of requests are short, routine and easy, and a minority are long, ambiguous and hard. Sending all of them to one model means the easy majority pays for the hard minority’s capability, on every call.

Routing is the practice of matching the model to the request. Three signals, in ascending order of reliability.

Context size. A 900-token request does not need a model chosen for its million-token window. Long-context models are typically priced and provisioned for that capability, and short requests get no benefit from it.

Task type. Classification, extraction, routing, short factual answers from provided passages: small models do these well. Multi-step reasoning, code generation, nuanced writing, ambiguous judgement: these are where a larger model earns its price.

Measured difficulty. The most reliable and the most work: run your evaluation set through both models and find the slice where the small one matches the large one. That slice, defined by whatever features distinguish it, is your routing rule.

The cascade

The alternative to routing up front is trying cheap first and escalating on failure.

out = small_model(req)
if not validates(out) or out.confidence == "low" or is_refusal(out):
    out = large_model(req)

This works when the check is cheap and reliable. Schema validation is both. A verifiable assertion (the cited quote appears in the passage, the extracted date parses, the total sums correctly) is both. A model’s own claim to be confident is neither, and building a cascade on self-reported confidence is how a cascade stops escalating the cases that most need it.

The arithmetic that decides whether it is worth it

A cascade is only cheaper if the cheap path handles most traffic. Suppose the small model is a twentieth the price, and the escalation rate is e:

cost_ratio = (1 + 20e) / 20

At e = 0.1 you pay 15% of the large-model cost. At e = 0.3 you pay 35%. At e = 0.7, 75%: most of the saving is gone and you have added a serial call to the majority of requests.

So measure the escalation rate before committing to the architecture. And measure it on real traffic rather than an evaluation set, because evaluation sets over-represent the hard cases somebody thought worth writing down.

The latency cost is paid by the wrong people

A cascade adds a full small-model call to every escalated request. Those are, by construction, the hard ones: the users with the most complicated questions get the slowest answers.

Two mitigations. Run the cheap and expensive calls in parallel for request types you know escalate often, and discard the loser; you pay for both and halve the latency, which is a legitimate trade for an interactive surface. Or route up front on features you can compute without a model call, which costs nothing and escalates nothing.

Routing has its own failure mode

A router is a classifier, and classifiers are wrong. A router that sends 8% of hard requests to the small model produces a system with an 8% floor of bad answers that no amount of prompt work removes, and the failures are invisible in aggregate because 92% of the traffic is fine.

So instrument the router as you would any classifier. Sample its decisions, check them, and track the rate at which the small model’s output fails validation as a proxy for misrouting. If that rate climbs, the traffic mix has shifted underneath the rule.

The free path

This is where locally hosted models fit into a production system honestly. An 8B model on a modest GPU, or a 3B on CPU, handles classification, routing, query rewriting and extraction competently, at zero marginal cost per call and with no data leaving your infrastructure.

On a hosted-only stack, the equivalent is the vendor’s smallest model, which is typically ten to thirty times cheaper than their largest.

Either way, the design principle is the same and it is the point of this lesson: the expensive model should be doing the work that requires it, and most of the work does not.

Start with one route

Do not build a routing framework. Find the single highest-volume, most routine request type in your traffic, move it to a small model, and measure quality on that slice for a fortnight.

If it holds, take the next one. Most of the available saving usually comes from the first two or three routes, and a system with three explicit routes is maintainable in a way that a learned router over eleven models is not.