{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"routing-by-context-size","title":"Not every request needs the big model","url":"https://addaly.com/learn/context-engineering/routing-by-context-size","position":47,"of":87,"minutes":8,"answer":"Match the model to the request (by context size, task type.","keyIdea":"Match the model to the request (by context size, task type, or a measured slice where a small model matches a large one) and remember that a cascade only saves money while the escalation rate stays low, so measure that rate on real traffic before building one."},"body":{"format":"markdown","text":"## The distribution is not flat\n\nLook at a day of traffic and you will usually find the same shape: a large majority of requests are short, routine and easy, and a minority are long, ambiguous and hard. Sending all of them to one model means the easy majority pays for the hard minority's capability, on every call.\n\nRouting is the practice of matching the model to the request. Three signals, in ascending order of reliability.\n\n**Context size.** A 900-token request does not need a model chosen for its million-token window. Long-context models are typically priced and provisioned for that capability, and short requests get no benefit from it.\n\n**Task type.** Classification, extraction, routing, short factual answers from provided passages: small models do these well. Multi-step reasoning, code generation, nuanced writing, ambiguous judgement: these are where a larger model earns its price.\n\n**Measured difficulty.** The most reliable and the most work: run your evaluation set through both models and find the slice where the small one matches the large one. That slice, defined by whatever features distinguish it, is your routing rule.\n\n## The cascade\n\nThe alternative to routing up front is trying cheap first and escalating on failure.\n\n```python\nout = small_model(req)\nif not validates(out) or out.confidence == \"low\" or is_refusal(out):\n    out = large_model(req)\n```\n\nThis works when the check is cheap and reliable. Schema validation is both. A verifiable assertion (the cited quote appears in the passage, the extracted date parses, the total sums correctly) is both. A model's own claim to be confident is neither, and building a cascade on self-reported confidence is how a cascade stops escalating the cases that most need it.\n\n## The arithmetic that decides whether it is worth it\n\nA cascade is only cheaper if the cheap path handles most traffic. Suppose the small model is a twentieth the price, and the escalation rate is `e`:\n\n```\ncost_ratio = (1 + 20e) / 20\n```\n\nAt `e = 0.1` you pay 15% of the large-model cost. At `e = 0.3` you pay 35%. At `e = 0.7`, 75%: most of the saving is gone and you have added a serial call to the majority of requests.\n\nSo measure the escalation rate before committing to the architecture. And measure it on real traffic rather than an evaluation set, because evaluation sets over-represent the hard cases somebody thought worth writing down.\n\n## The latency cost is paid by the wrong people\n\nA cascade adds a full small-model call to every escalated request. Those are, by construction, the hard ones: the users with the most complicated questions get the slowest answers.\n\nTwo mitigations. Run the cheap and expensive calls in parallel for request types you know escalate often, and discard the loser; you pay for both and halve the latency, which is a legitimate trade for an interactive surface. Or route up front on features you can compute without a model call, which costs nothing and escalates nothing.\n\n## Routing has its own failure mode\n\nA router is a classifier, and classifiers are wrong. A router that sends 8% of hard requests to the small model produces a system with an 8% floor of bad answers that no amount of prompt work removes, and the failures are invisible in aggregate because 92% of the traffic is fine.\n\nSo instrument the router as you would any classifier. Sample its decisions, check them, and track the rate at which the small model's output fails validation as a proxy for misrouting. If that rate climbs, the traffic mix has shifted underneath the rule.\n\n## The free path\n\nThis is where locally hosted models fit into a production system honestly. An 8B model on a modest GPU, or a 3B on CPU, handles classification, routing, query rewriting and extraction competently, at zero marginal cost per call and with no data leaving your infrastructure.\n\nOn a hosted-only stack, the equivalent is the vendor's smallest model, which is typically ten to thirty times cheaper than their largest.\n\nEither way, the design principle is the same and it is the point of this lesson: the expensive model should be doing the work that requires it, and most of the work does not.\n\n## Start with one route\n\nDo not build a routing framework. Find the single highest-volume, most routine request type in your traffic, move it to a small model, and measure quality on that slice for a fortnight.\n\nIf it holds, take the next one. Most of the available saving usually comes from the first two or three routes, and a system with three explicit routes is maintainable in a way that a learned router over eleven models is not."},"check":{"question":"A cascade uses a model that is one twentieth the price, escalating when the cheap output fails validation. The escalation rate turns out to be 70%. What follows?","options":[{"text":"The cascade is working as intended, since the validator is catching most weak outputs.","correct":false,"tempting":"Treats a high catch rate as success when it means the cheap path is unsuited to the traffic."},{"text":"The validator is too strict and should be loosened until the escalation rate falls.","correct":false,"tempting":"Fixes the metric by weakening the check, admitting exactly the bad outputs the cascade exists to catch."},{"text":"Most of the saving is gone and every escalated request now carries an extra serial call.","correct":true},{"text":"The small model needs a larger context budget, since validation failures usually indicate truncated input.","correct":false,"tempting":"Assumes a cause the escalation rate does not indicate; validation failures more often reflect capability than budget."}]},"prev":{"slug":"batching-and-the-slow-lane","title":"Work that can wait","url":"https://addaly.com/learn/context-engineering/batching-and-the-slow-lane"},"next":{"slug":"denial-of-wallet-and-context-bombs","title":"How one user spends your whole budget","url":"https://addaly.com/learn/context-engineering/denial-of-wallet-and-context-bombs"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Not every request needs the big model', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/routing-by-context-size (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Not every request needs the big model","courseTitle":"Context Engineering","module":{"index":5,"slug":"cost-cache-and-latency","title":"Cost, caching and latency"},"answer":null,"keyIdea":"Match the model to the request (by context size, task type, or a measured slice where a small model matches a large one) and remember that a cascade only saves money while the escalation rate stays low, so measure that rate on real traffic before building one.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-distribution-is-not-flat","runs":[{"text":"The distribution is not flat"}]},{"t":"p","runs":[{"text":"Look at a day of traffic and you will usually find the same shape: a large majority of requests are short, routine and easy, and a minority are long, ambiguous and hard. Sending all of them to one model means the easy majority pays for the hard minority’s capability, on every call."}]},{"t":"p","runs":[{"text":"Routing is the practice of matching the model to the request. Three signals, in ascending order of reliability."}]},{"t":"p","runs":[{"text":"Context size.","b":true},{"text":" A 900-token request does not need a model chosen for its million-token window. Long-context models are typically priced and provisioned for that capability, and short requests get no benefit from it."}]},{"t":"p","runs":[{"text":"Task type.","b":true},{"text":" Classification, extraction, routing, short factual answers from provided passages: small models do these well. Multi-step reasoning, code generation, nuanced writing, ambiguous judgement: these are where a larger model earns its price."}]},{"t":"p","runs":[{"text":"Measured difficulty.","b":true},{"text":" The most reliable and the most work: run your evaluation set through both models and find the slice where the small one matches the large one. That slice, defined by whatever features distinguish it, is your routing rule."}]},{"t":"h2","id":"the-cascade","runs":[{"text":"The cascade"}]},{"t":"p","runs":[{"text":"The alternative to routing up front is trying cheap first and escalating on failure."}]},{"t":"code","lang":"python","label":"Python","text":"out = small_model(req)\nif not validates(out) or out.confidence == \"low\" or is_refusal(out):\n    out = large_model(req)"},{"t":"p","runs":[{"text":"This works when the check is cheap and reliable. Schema validation is both. A verifiable assertion (the cited quote appears in the passage, the extracted date parses, the total sums correctly) is both. A model’s own claim to be confident is neither, and building a cascade on self-reported confidence is how a cascade stops escalating the cases that most need it."}]},{"t":"h2","id":"the-arithmetic-that-decides-whether-it-is-worth-it","runs":[{"text":"The arithmetic that decides whether it is worth it"}]},{"t":"p","runs":[{"text":"A cascade is only cheaper if the cheap path handles most traffic. Suppose the small model is a twentieth the price, and the escalation rate is "},{"text":"e","code":true},{"text":":"}]},{"t":"code","lang":"","label":"Text","text":"cost_ratio = (1 + 20e) / 20"},{"t":"p","runs":[{"text":"At "},{"text":"e = 0.1","code":true},{"text":" you pay 15% of the large-model cost. At "},{"text":"e = 0.3","code":true},{"text":" you pay 35%. At "},{"text":"e = 0.7","code":true},{"text":", 75%: most of the saving is gone and you have added a serial call to the majority of requests."}]},{"t":"p","runs":[{"text":"So measure the escalation rate before committing to the architecture. And measure it on real traffic rather than an evaluation set, because evaluation sets over-represent the hard cases somebody thought worth writing down."}]},{"t":"h2","id":"the-latency-cost-is-paid-by-the-wrong-people","runs":[{"text":"The latency cost is paid by the wrong people"}]},{"t":"p","runs":[{"text":"A cascade adds a full small-model call to every escalated request. Those are, by construction, the hard ones: the users with the most complicated questions get the slowest answers."}]},{"t":"p","runs":[{"text":"Two mitigations. Run the cheap and expensive calls in parallel for request types you know escalate often, and discard the loser; you pay for both and halve the latency, which is a legitimate trade for an interactive surface. Or route up front on features you can compute without a model call, which costs nothing and escalates nothing."}]},{"t":"h2","id":"routing-has-its-own-failure-mode","runs":[{"text":"Routing has its own failure mode"}]},{"t":"p","runs":[{"text":"A router is a classifier, and classifiers are wrong. A router that sends 8% of hard requests to the small model produces a system with an 8% floor of bad answers that no amount of prompt work removes, and the failures are invisible in aggregate because 92% of the traffic is fine."}]},{"t":"p","runs":[{"text":"So instrument the router as you would any classifier. Sample its decisions, check them, and track the rate at which the small model’s output fails validation as a proxy for misrouting. If that rate climbs, the traffic mix has shifted underneath the rule."}]},{"t":"h2","id":"the-free-path","runs":[{"text":"The free path"}],"role":"free"},{"t":"p","runs":[{"text":"This is where locally hosted models fit into a production system honestly. An 8B model on a modest GPU, or a 3B on CPU, handles classification, routing, query rewriting and extraction competently, at zero marginal cost per call and with no data leaving your infrastructure."}]},{"t":"p","runs":[{"text":"On a hosted-only stack, the equivalent is the vendor’s smallest model, which is typically ten to thirty times cheaper than their largest."}]},{"t":"p","runs":[{"text":"Either way, the design principle is the same and it is the point of this lesson: the expensive model should be doing the work that requires it, and most of the work does not."}]},{"t":"h2","id":"start-with-one-route","runs":[{"text":"Start with one route"}]},{"t":"p","runs":[{"text":"Do not build a routing framework. Find the single highest-volume, most routine request type in your traffic, move it to a small model, and measure quality on that slice for a fortnight."}]},{"t":"p","runs":[{"text":"If it holds, take the next one. Most of the available saving usually comes from the first two or three routes, and a system with three explicit routes is maintainable in a way that a learned router over eleven models is not."}]}],"check":{"question":[{"text":"A cascade uses a model that is one twentieth the price, escalating when the cheap output fails validation. The escalation rate turns out to be 70%. What follows?"}],"options":[{"text":[{"text":"The cascade is working as intended, since the validator is catching most weak outputs."}],"correct":false,"tempting":[{"text":"Treats a high catch rate as success when it means the cheap path is unsuited to the traffic."}]},{"text":[{"text":"The validator is too strict and should be loosened until the escalation rate falls."}],"correct":false,"tempting":[{"text":"Fixes the metric by weakening the check, admitting exactly the bad outputs the cascade exists to catch."}]},{"text":[{"text":"Most of the saving is gone and every escalated request now carries an extra serial call."}],"correct":true,"tempting":null},{"text":[{"text":"The small model needs a larger context budget, since validation failures usually indicate truncated input."}],"correct":false,"tempting":[{"text":"Assumes a cause the escalation rate does not indicate; validation failures more often reflect capability than budget."}]}],"order":[1,3,0,2],"why":null},"prev":{"slug":"batching-and-the-slow-lane","title":"Work that can wait","minutes":8},"next":{"slug":"denial-of-wallet-and-context-bombs","title":"How one user spends your whole budget","minutes":9}}}