{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"cascades-and-routing","title":"Cascades and routers: paying for the hard cases only","url":"https://addaly.com/learn/how-llms-work/cascades-and-routing","position":67,"of":78,"minutes":8,"answer":"A cascade is only as good as the signal that decides to escalate, so prefer a deterministic check, then a calibrated score, then consistency.","keyIdea":"A cascade is only as good as the signal that decides to escalate, so prefer a deterministic check, then a calibrated score, then consistency, and measure the routed system by re-running a sample of cheap-routed requests through the expensive model."},"body":{"format":"markdown","text":"## The shape of a workload\n\nLook at a day of real requests and most of them are easy: a reformat, a short classification, a question the small model answers as well as the large one. A few are hard. Sending everything to the frontier model pays the frontier price on every easy request in order to be ready for the hard ones, and the previous lessons' arithmetic says that price is a multiple of ten or more.\n\nPublished routing studies, on mixed workloads, report that sending a third to two-thirds of requests to a cheap model kept around 95 per cent of the strong model's measured quality at somewhere between a third and a half of the cost. The exact figures depend on the workload; the shape does not. There are two ways to build this, and each has a weak point you can name in advance.\n\n## Cascade: try cheap, escalate on doubt\n\nThe cheap model answers first. Something then decides whether to accept the answer or send the request on to the expensive model. Cost per request is the cheap price plus the escalation rate times the expensive price; latency in the worst case is both calls in sequence.\n\nThe whole design rests on the deciding signal, and module six told you what to expect from it. A small model's verbalised confidence clusters high whatever the answer. Its log-probabilities rank but do not calibrate. Semantic entropy works and costs several samples. So the order of preference is:\n\n1. **A deterministic check, when one exists.** The JSON validates. The code passes the tests. The extracted date parses. The answer is one of the allowed labels. These are free and exact, and a task that admits one should be structured to admit one: module three's constrained decoding is often what makes it possible.\n2. **A calibrated score.** The uncertainty lesson's procedure, applied to the cheap model on labelled traffic: bucket its scores, measure the accuracy per bucket, and set the escalation threshold where the accuracy drops below what you will tolerate.\n3. **Consistency.** Sample the cheap model three times; disagreement escalates. Costs three cheap calls, which is often still far below one expensive one.\n\nThe classic failure is a threshold set by feel on an uncalibrated signal. A cascade that escalates four per cent of requests on \"confidence under 70\" is almost always a cascade whose small model is rarely under 70 about anything, including its mistakes.\n\n## Router: decide first, call once\n\nThe alternative sends each request to one model, chosen before any answer exists. The chooser is a small classifier (a fine-tuned encoder of the kind module one described, or a nearest-neighbour lookup over embeddings of past requests) trained on your own traffic: for each labelled item, which model got it right at the lowest cost.\n\nA router has no worst-case double latency and no dependence on the cheap model's self-assessment. It has its own weak points. It needs a few thousand labelled examples from your real distribution. Its accuracy decays as the traffic changes (module six's input drift) so it is retrained on a schedule. And its errors are silent: a hard request routed cheap gets a confident wrong answer, with nothing downstream to catch it.\n\n## Routing without machine learning\n\nBefore either of the above, try rules. Route by language (the Hindi lesson's point about which models handle which languages) by input length, by the presence of code, by the customer tier, by whether the request touches a tool. In many workloads a handful of rules captures most of the saving, is auditable, and never drifts in a way you cannot see in the code. Add the learned router only for the residual that rules cannot separate.\n\n## Composing with the rest of the module\n\nCaching sits before routing: a cached prefix is cheap at either model. The batch endpoint is a lane of its own for anything that can wait, whatever its difficulty. Constrained decoding on the cheap model makes its output checkable, which turns a cascade's weakest link into its strongest. And the cheap model can be a local one from this module's fourth lesson, at zero marginal cost, which changes the arithmetic of what counts as easy enough.\n\nOne caution from the prompt-sensitivity lesson: two models rarely want the same prompt. A routed system has at least two prompts to maintain and evaluate, and a change to one is not a change to the other.\n\n## Measuring a routed system\n\nMeasure the system, not the models. A reported \"95 per cent of quality retained\" is an average that hides which five per cent was lost, and it is usually the hardest, most valuable requests. Keep a random sample of requests that were routed cheap, re-run them through the expensive model, and compare, continuously, not once. That sample is the only instrument that tells you whether the router is still doing its job after the traffic moved.\n\n## The honest state\n\nFully automatic routers are still fragile. The version that holds up in production is a few deterministic rules, one calibrated uncertainty gate, a cheap deterministic check wherever the task allows one, and the continuous sample above. Providers have begun routing inside their own products (a request to one model name may be served by several) which is one more mechanism behind module six's \"same name, different behaviour\", and one more reason to keep the canary set running."},"check":{"question":"A cascade sends every request to a cheap model first and escalates to an expensive one when the cheap model's stated confidence is below 70 out of 100. Only 4 per cent of requests escalate and overall quality is well below the expensive model's. What is the likely cause?","options":[{"text":"Verbalised confidence clusters high regardless of correctness, so few wrong answers fall under the threshold.","correct":true},{"text":"The threshold of 70 is too high; lowering it to 50 would let the cheap model handle its confident cases while escalating the rest.","correct":false,"tempting":"Wrong direction: a lower threshold escalates even fewer requests, and the problem is that the signal does not separate right from wrong at any threshold."},{"text":"The cascade is inverted: the expensive model should answer first and the cheap model should be used only to check its output.","correct":false,"tempting":"That design pays the expensive price on every request, which removes the saving the cascade exists to make; the fault is in the gate, not the order."},{"text":"Four per cent is too low an escalation rate for any cascade to function, because the expensive model needs a steady stream of requests to keep its prompt cache warm and its batches full.","correct":false,"tempting":"Borrows real serving mechanisms to invent a constraint; the expensive model's cache and batching affect its cost and latency, not the correctness of what the cheap model let through."}]},"prev":{"slug":"distillation-and-small-models","title":"Distillation: how small models got good","url":"https://addaly.com/learn/how-llms-work/distillation-and-small-models"},"next":{"slug":"beyond-the-transformer","title":"Beyond the transformer: what the alternatives trade","url":"https://addaly.com/learn/how-llms-work/beyond-the-transformer"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Cascades and routers: paying for the hard cases only', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/cascades-and-routing (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Cascades and routers: paying for the hard cases only","courseTitle":"How a Language Model Actually Works","module":{"index":7,"slug":"running-it","title":"Running it: memory, speed and money"},"answer":null,"keyIdea":"A cascade is only as good as the signal that decides to escalate, so prefer a deterministic check, then a calibrated score, then consistency, and measure the routed system by re-running a sample of cheap-routed requests through the expensive model.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-shape-of-a-workload","runs":[{"text":"The shape of a workload"}]},{"t":"p","runs":[{"text":"Look at a day of real requests and most of them are easy: a reformat, a short classification, a question the small model answers as well as the large one. A few are hard. Sending everything to the frontier model pays the frontier price on every easy request in order to be ready for the hard ones, and the previous lessons’ arithmetic says that price is a multiple of ten or more."}]},{"t":"p","runs":[{"text":"Published routing studies, on mixed workloads, report that sending a third to two-thirds of requests to a cheap model kept around 95 per cent of the strong model’s measured quality at somewhere between a third and a half of the cost. The exact figures depend on the workload; the shape does not. There are two ways to build this, and each has a weak point you can name in advance."}]},{"t":"h2","id":"cascade-try-cheap-escalate-on-doubt","runs":[{"text":"Cascade: try cheap, escalate on doubt"}]},{"t":"p","runs":[{"text":"The cheap model answers first. Something then decides whether to accept the answer or send the request on to the expensive model. Cost per request is the cheap price plus the escalation rate times the expensive price; latency in the worst case is both calls in sequence."}]},{"t":"p","runs":[{"text":"The whole design rests on the deciding signal, and module six told you what to expect from it. A small model’s verbalised confidence clusters high whatever the answer. Its log-probabilities rank but do not calibrate. Semantic entropy works and costs several samples. So the order of preference is:"}]},{"t":"ol","start":1,"items":[[{"text":"A deterministic check, when one exists.","b":true},{"text":" The JSON validates. The code passes the tests. The extracted date parses. The answer is one of the allowed labels. These are free and exact, and a task that admits one should be structured to admit one: module three’s constrained decoding is often what makes it possible."}],[{"text":"A calibrated score.","b":true},{"text":" The uncertainty lesson’s procedure, applied to the cheap model on labelled traffic: bucket its scores, measure the accuracy per bucket, and set the escalation threshold where the accuracy drops below what you will tolerate."}],[{"text":"Consistency.","b":true},{"text":" Sample the cheap model three times; disagreement escalates. Costs three cheap calls, which is often still far below one expensive one."}]]},{"t":"p","runs":[{"text":"The classic failure is a threshold set by feel on an uncalibrated signal. A cascade that escalates four per cent of requests on “confidence under 70” is almost always a cascade whose small model is rarely under 70 about anything, including its mistakes."}]},{"t":"h2","id":"router-decide-first-call-once","runs":[{"text":"Router: decide first, call once"}]},{"t":"p","runs":[{"text":"The alternative sends each request to one model, chosen before any answer exists. The chooser is a small classifier (a fine-tuned encoder of the kind module one described, or a nearest-neighbour lookup over embeddings of past requests) trained on your own traffic: for each labelled item, which model got it right at the lowest cost."}]},{"t":"p","runs":[{"text":"A router has no worst-case double latency and no dependence on the cheap model’s self-assessment. It has its own weak points. It needs a few thousand labelled examples from your real distribution. Its accuracy decays as the traffic changes (module six’s input drift) so it is retrained on a schedule. And its errors are silent: a hard request routed cheap gets a confident wrong answer, with nothing downstream to catch it."}]},{"t":"h2","id":"routing-without-machine-learning","runs":[{"text":"Routing without machine learning"}]},{"t":"p","runs":[{"text":"Before either of the above, try rules. Route by language (the Hindi lesson’s point about which models handle which languages) by input length, by the presence of code, by the customer tier, by whether the request touches a tool. In many workloads a handful of rules captures most of the saving, is auditable, and never drifts in a way you cannot see in the code. Add the learned router only for the residual that rules cannot separate."}]},{"t":"h2","id":"composing-with-the-rest-of-the-module","runs":[{"text":"Composing with the rest of the module"}]},{"t":"p","runs":[{"text":"Caching sits before routing: a cached prefix is cheap at either model. The batch endpoint is a lane of its own for anything that can wait, whatever its difficulty. Constrained decoding on the cheap model makes its output checkable, which turns a cascade’s weakest link into its strongest. And the cheap model can be a local one from this module’s fourth lesson, at zero marginal cost, which changes the arithmetic of what counts as easy enough."}]},{"t":"p","runs":[{"text":"One caution from the prompt-sensitivity lesson: two models rarely want the same prompt. A routed system has at least two prompts to maintain and evaluate, and a change to one is not a change to the other."}]},{"t":"h2","id":"measuring-a-routed-system","runs":[{"text":"Measuring a routed system"}]},{"t":"p","runs":[{"text":"Measure the system, not the models. A reported “95 per cent of quality retained” is an average that hides which five per cent was lost, and it is usually the hardest, most valuable requests. Keep a random sample of requests that were routed cheap, re-run them through the expensive model, and compare, continuously, not once. That sample is the only instrument that tells you whether the router is still doing its job after the traffic moved."}]},{"t":"h2","id":"the-honest-state","runs":[{"text":"The honest state"}]},{"t":"p","runs":[{"text":"Fully automatic routers are still fragile. The version that holds up in production is a few deterministic rules, one calibrated uncertainty gate, a cheap deterministic check wherever the task allows one, and the continuous sample above. Providers have begun routing inside their own products (a request to one model name may be served by several) which is one more mechanism behind module six’s “same name, different behaviour”, and one more reason to keep the canary set running."}]}],"check":{"question":[{"text":"A cascade sends every request to a cheap model first and escalates to an expensive one when the cheap model’s stated confidence is below 70 out of 100. Only 4 per cent of requests escalate and overall quality is well below the expensive model’s. What is the likely cause?"}],"options":[{"text":[{"text":"Verbalised confidence clusters high regardless of correctness, so few wrong answers fall under the threshold."}],"correct":true,"tempting":null},{"text":[{"text":"The threshold of 70 is too high; lowering it to 50 would let the cheap model handle its confident cases while escalating the rest."}],"correct":false,"tempting":[{"text":"Wrong direction: a lower threshold escalates even fewer requests, and the problem is that the signal does not separate right from wrong at any threshold."}]},{"text":[{"text":"The cascade is inverted: the expensive model should answer first and the cheap model should be used only to check its output."}],"correct":false,"tempting":[{"text":"That design pays the expensive price on every request, which removes the saving the cascade exists to make; the fault is in the gate, not the order."}]},{"text":[{"text":"Four per cent is too low an escalation rate for any cascade to function, because the expensive model needs a steady stream of requests to keep its prompt cache warm and its batches full."}],"correct":false,"tempting":[{"text":"Borrows real serving mechanisms to invent a constraint; the expensive model’s cache and batching affect its cost and latency, not the correctness of what the cheap model let through."}]}],"order":[2,1,3,0],"why":null},"prev":{"slug":"distillation-and-small-models","title":"Distillation: how small models got good","minutes":9},"next":{"slug":"beyond-the-transformer","title":"Beyond the transformer: what the alternatives trade","minutes":9}}}