{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"routing-and-cascades","title":"Routing and cascades: paying the high price only where it is needed","url":"https://addaly.com/learn/building-with-ai/routing-and-cascades","position":75,"of":80,"minutes":9,"answer":"Most requests are easy and a cheap model handles them, so a router that sends easy traffic to the small model and escalates on a measurable signal (a failed validation, a refusal, low judge confidence.","keyIdea":"Most requests are easy and a cheap model handles them, so a router that sends easy traffic to the small model and escalates on a measurable signal (a failed validation, a refusal, low judge confidence, a hard-intent label) cuts cost by more than half, provided the escalation rule actually fires and each route is evaluated on its own slice."},"body":{"format":"markdown","text":"## The shape of real traffic\n\nLook at a week of requests to any assistant and most of them are simple: a factual question with a clear answer in the corpus, a greeting, a request to rephrase something. A small fraction are hard: multi-step reasoning, an ambiguous question, a long document, a tool sequence. The frontier model earns its price on the hard fraction and wastes it on the rest. Routing is sending each request to the cheapest model that handles it, and it is the largest cost lever after the batch window in most products.\n\n## Two designs\n\n**A router decides up front.** A classifier looks at the request and picks the model before any generation. The classifier can be heuristics (input length, presence of an attachment, a tool-requiring intent, a language the small model handles poorly) or a tiny model asked to label the request as easy or hard, or the intent label you already compute for analytics. It costs a few milliseconds and a fraction of a cent.\n\n**A cascade escalates on failure.** The small model answers first. If the answer fails a check (the structured output did not validate, the model refused, the citation check failed, a cheap judge scored it low, the reranker found nothing) the request is re-run on the large model. The small model's attempt is wasted on escalated requests, so the cascade pays extra on the hard fraction and saves on the easy one.\n\nThe router is better when hardness is predictable from the input; the cascade is better when it is only visible in the output. Many systems use both: route the obvious cases, cascade the rest.\n\n## The arithmetic\n\nSuppose the large model costs ten times the small one per token, and seventy per cent of requests can be handled by the small model. Sending everything to the large model costs 100 units. Routing: 70 requests at 1 unit and 30 at 10 units is 370 units, against 1,000 for all-large: a 63 per cent saving. A cascade that escalates 30 per cent pays 100 small attempts plus 300 large: 400 units, a 60 per cent saving. If the small model handles ninety per cent, the saving approaches ninety per cent. The number that decides it is the easy fraction, and only your evaluation set, run on both models, tells you what it is.\n\n## The two failure modes\n\n**The escalation never fires.** A cascade whose check is \"did the small model return something?\" escalates nothing, because the small model always returns something. Wrong answers ship at the low price. The check must be able to fail on a wrong answer (validation, citation, a judge) and the rate at which it fires should be reported daily. A rate near zero is a rule that is not working.\n\n**The escalation always fires.** A check that is too strict sends everything to the large model after paying for a small attempt first, and the cascade costs more than no cascade at all. Watch the escalation rate; if it climbs past the easy fraction the evaluation set predicted, the check or the small model has changed.\n\n## Evaluate per route\n\nThe harness must know which route each case took. Report pass rates for the small-model slice and the large-model slice separately, and for the escalated cases before and after escalation. An overall pass rate hides a small model that is failing quietly on a slice the router marks easy. Tag the evaluation cases by expected route, and check that the router's decisions match the tags: a router that sends a case tagged hard to the small model is a finding.\n\n## Behaviour must match across routes\n\nUsers notice when the assistant changes voice between questions. The two models are given the same system prompt and the same examples, and the evaluation set includes checks on tone and format that both must pass. Where the small model cannot match (a formatting quirk, a weaker grasp of one language) either fix it with a few-shot example for that model or route that slice to the large one.\n\n## Where the small model wins outright\n\nSometimes the small model, with good retrieval, beats the large model on the evaluation set for the easy slice, on speed and on quality both, because the task is a lookup and the large model's extra capacity is spent on elaboration nobody asked for. Do not assume the expensive model is better for every slice; run the set and read the numbers per slice.\n\n## Keep it changeable\n\nModel names, thresholds and the routing rule live in configuration behind the same flags as prompts, from the shipping lesson. The models will change, the price ratio between them will change, and the easy fraction will change as the product does. A routing rule from six months ago, hard-coded, is spending money on a ratio that no longer exists."},"check":{"question":"A team adds a cascade: a small model answers first and the large model is used if the small model \"fails\". Costs fall by 80 per cent, but the evaluation pass rate also falls sharply. What is the most likely cause?","options":[{"text":"The small model is too weak for any of the traffic and should be removed from the cascade.","correct":false,"tempting":"Contradicted by the cost drop, which shows it is handling most requests; the problem is what happens when it handles them badly."},{"text":"The escalation check cannot fail on a wrong answer, so incorrect small-model responses ship without escalation.","correct":true},{"text":"The large model is being invoked too often, adding its errors on top of the small model's.","correct":false,"tempting":"Frequent escalation would raise costs, not cut them by 80 per cent, and the large model rarely adds errors."},{"text":"The two models use different system prompts, so the evaluation set's format checks fail on the small model's output.","correct":false,"tempting":"Would produce format failures, not a general drop in correctness, and is fixable without touching the cascade logic."}]},"prev":{"slug":"the-batch-window","title":"The batch window: half price for anything that can wait","url":"https://addaly.com/learn/building-with-ai/the-batch-window"},"next":{"slug":"running-your-own-inference","title":"Running your own inference: the break-even arithmetic","url":"https://addaly.com/learn/building-with-ai/running-your-own-inference"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Routing and cascades: paying the high price only where it is needed', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/routing-and-cascades (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Routing and cascades: paying the high price only where it is needed","courseTitle":"Building with AI","module":{"index":8,"slug":"cost-latency-and-running-it","title":"Cost, latency and running it in production"},"answer":null,"keyIdea":"Most requests are easy and a cheap model handles them, so a router that sends easy traffic to the small model and escalates on a measurable signal (a failed validation, a refusal, low judge confidence, a hard-intent label) cuts cost by more than half, provided the escalation rule actually fires and each route is evaluated on its own slice.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-shape-of-real-traffic","runs":[{"text":"The shape of real traffic"}]},{"t":"p","runs":[{"text":"Look at a week of requests to any assistant and most of them are simple: a factual question with a clear answer in the corpus, a greeting, a request to rephrase something. A small fraction are hard: multi-step reasoning, an ambiguous question, a long document, a tool sequence. The frontier model earns its price on the hard fraction and wastes it on the rest. Routing is sending each request to the cheapest model that handles it, and it is the largest cost lever after the batch window in most products."}]},{"t":"h2","id":"two-designs","runs":[{"text":"Two designs"}]},{"t":"p","runs":[{"text":"A router decides up front.","b":true},{"text":" A classifier looks at the request and picks the model before any generation. The classifier can be heuristics (input length, presence of an attachment, a tool-requiring intent, a language the small model handles poorly) or a tiny model asked to label the request as easy or hard, or the intent label you already compute for analytics. It costs a few milliseconds and a fraction of a cent."}]},{"t":"p","runs":[{"text":"A cascade escalates on failure.","b":true},{"text":" The small model answers first. If the answer fails a check (the structured output did not validate, the model refused, the citation check failed, a cheap judge scored it low, the reranker found nothing) the request is re-run on the large model. The small model’s attempt is wasted on escalated requests, so the cascade pays extra on the hard fraction and saves on the easy one."}]},{"t":"p","runs":[{"text":"The router is better when hardness is predictable from the input; the cascade is better when it is only visible in the output. Many systems use both: route the obvious cases, cascade the rest."}]},{"t":"h2","id":"the-arithmetic","runs":[{"text":"The arithmetic"}]},{"t":"p","runs":[{"text":"Suppose the large model costs ten times the small one per token, and seventy per cent of requests can be handled by the small model. Sending everything to the large model costs 100 units. Routing: 70 requests at 1 unit and 30 at 10 units is 370 units, against 1,000 for all-large: a 63 per cent saving. A cascade that escalates 30 per cent pays 100 small attempts plus 300 large: 400 units, a 60 per cent saving. If the small model handles ninety per cent, the saving approaches ninety per cent. The number that decides it is the easy fraction, and only your evaluation set, run on both models, tells you what it is."}]},{"t":"h2","id":"the-two-failure-modes","runs":[{"text":"The two failure modes"}]},{"t":"p","runs":[{"text":"The escalation never fires.","b":true},{"text":" A cascade whose check is “did the small model return something?” escalates nothing, because the small model always returns something. Wrong answers ship at the low price. The check must be able to fail on a wrong answer (validation, citation, a judge) and the rate at which it fires should be reported daily. A rate near zero is a rule that is not working."}]},{"t":"p","runs":[{"text":"The escalation always fires.","b":true},{"text":" A check that is too strict sends everything to the large model after paying for a small attempt first, and the cascade costs more than no cascade at all. Watch the escalation rate; if it climbs past the easy fraction the evaluation set predicted, the check or the small model has changed."}]},{"t":"h2","id":"evaluate-per-route","runs":[{"text":"Evaluate per route"}]},{"t":"p","runs":[{"text":"The harness must know which route each case took. Report pass rates for the small-model slice and the large-model slice separately, and for the escalated cases before and after escalation. An overall pass rate hides a small model that is failing quietly on a slice the router marks easy. Tag the evaluation cases by expected route, and check that the router’s decisions match the tags: a router that sends a case tagged hard to the small model is a finding."}]},{"t":"h2","id":"behaviour-must-match-across-routes","runs":[{"text":"Behaviour must match across routes"}]},{"t":"p","runs":[{"text":"Users notice when the assistant changes voice between questions. The two models are given the same system prompt and the same examples, and the evaluation set includes checks on tone and format that both must pass. Where the small model cannot match (a formatting quirk, a weaker grasp of one language) either fix it with a few-shot example for that model or route that slice to the large one."}]},{"t":"h2","id":"where-the-small-model-wins-outright","runs":[{"text":"Where the small model wins outright"}]},{"t":"p","runs":[{"text":"Sometimes the small model, with good retrieval, beats the large model on the evaluation set for the easy slice, on speed and on quality both, because the task is a lookup and the large model’s extra capacity is spent on elaboration nobody asked for. Do not assume the expensive model is better for every slice; run the set and read the numbers per slice."}]},{"t":"h2","id":"keep-it-changeable","runs":[{"text":"Keep it changeable"}]},{"t":"p","runs":[{"text":"Model names, thresholds and the routing rule live in configuration behind the same flags as prompts, from the shipping lesson. The models will change, the price ratio between them will change, and the easy fraction will change as the product does. A routing rule from six months ago, hard-coded, is spending money on a ratio that no longer exists."}]}],"check":{"question":[{"text":"A team adds a cascade: a small model answers first and the large model is used if the small model “fails”. Costs fall by 80 per cent, but the evaluation pass rate also falls sharply. What is the most likely cause?"}],"options":[{"text":[{"text":"The small model is too weak for any of the traffic and should be removed from the cascade."}],"correct":false,"tempting":[{"text":"Contradicted by the cost drop, which shows it is handling most requests; the problem is what happens when it handles them badly."}]},{"text":[{"text":"The escalation check cannot fail on a wrong answer, so incorrect small-model responses ship without escalation."}],"correct":true,"tempting":null},{"text":[{"text":"The large model is being invoked too often, adding its errors on top of the small model’s."}],"correct":false,"tempting":[{"text":"Frequent escalation would raise costs, not cut them by 80 per cent, and the large model rarely adds errors."}]},{"text":[{"text":"The two models use different system prompts, so the evaluation set’s format checks fail on the small model’s output."}],"correct":false,"tempting":[{"text":"Would produce format failures, not a general drop in correctness, and is fixable without touching the cascade logic."}]}],"order":[0,2,3,1],"why":null},"prev":{"slug":"the-batch-window","title":"The batch window: half price for anything that can wait","minutes":8},"next":{"slug":"running-your-own-inference","title":"Running your own inference: the break-even arithmetic","minutes":9}}}