{"course":{"slug":"choosing-your-tools","title":"Choosing and Using the Tools","url":"https://addaly.com/learn/choosing-your-tools","level":"beginner","track":"work"},"lesson":{"slug":"routing-by-difficulty","title":"Spending money where it matters","url":"https://addaly.com/learn/choosing-your-tools/routing-by-difficulty","position":34,"of":76,"minutes":8,"answer":"A cascade (cheap model everywhere, expensive model only on flagged cases) commonly cuts cost by four fifths at equal or better quality.","keyIdea":"A cascade (cheap model everywhere, expensive model only on flagged cases) commonly cuts cost by four fifths at equal or better quality, provided you have measured on a hundred items which cases the cheap model actually gets wrong."},"body":{"format":"markdown","text":"## The shape production systems actually have\n\nPeople assume the systems they admire run the best available model on every request. Very few do. The common design is a **cascade**: a cheap model handles everything, and only the cases it cannot settle go to an expensive one.\n\n```\n1,000 items\n  -> small model handles all 1,000            $0.40\n  -> 120 flagged uncertain go to large model  $2.60\n                                              -----\n                                              $3.00\n  versus all 1,000 on the large model:       $21.60\n```\n\nEighty-six per cent saved, with the hard cases still getting the expensive treatment. This is not a compromise design. On many tasks it also scores *better* than the large model alone, because the escalation step forces a second look at exactly the items most likely to be wrong.\n\n## Deciding what escalates\n\nYou need a signal for \"this one is difficult\", and there are four that work without any special infrastructure.\n\n**Ask for a confidence word.** Have the small model end with `CONFIDENT` or `UNSURE`. It is not calibrated and it is not a probability, and it still correlates usefully with correctness: well enough to route on. Check the correlation on fifty items before trusting it.\n\n**Rules you already know.** Some cases are structurally hard: a document over thirty pages, a ticket from a top-tier customer, an invoice above a threshold, a message in a language the small model handles poorly. Route those up without asking anything.\n\n**Disagreement.** Run the small model twice at a non-zero temperature. If the two answers differ, escalate. Cheap, crude, and effective on classification.\n\n**Cost of being wrong.** The most important one, and it is a business decision rather than a technical one. Anything where a mistake costs money, safety or a customer goes to the expensive model regardless of how confident anything is.\n\n## The measurement that makes it honest\n\nA cascade is only correct if the small model is adequate on the easy cases, and the only way to know is to check.\n\nTake 100 items. Run both models. Compare. You will get four groups, and the interesting one is the fourth:\n\n- Both right: the cascade is safe here.\n- Both wrong: the model is not the problem; the prompt or the task is.\n- Large right, small wrong: this is what you are paying the large model for. How many, and do your escalation rules catch them?\n- **Small right, large wrong.** This group is always bigger than people expect. Large models are not uniformly better; they are better on average, and they overthink simple cases.\n\nIf your escalation rules catch most of the third group, the cascade is working. If they do not, tighten the rules rather than abandoning the design.\n\n## Where cascades are the wrong idea\n\n**When latency matters more than money.** Two model calls take twice as long. For anything a person is waiting on, that can be the wrong trade.\n\n**When the volume is small.** A hundred items a day at a cent each is a dollar. Engineering a cascade to save 80% of a dollar is not a good use of an afternoon.\n\n**When you cannot measure.** A cascade without the 100-item comparison is a guess that quietly ships worse work. If you are not going to check, use the model you trust.\n\n> Figure: A hundred items, run through both models. A two by two grid, rows Small model right and Small model wrong, columns Large model right and Large model wrong: Small model right and Large model right: 71; Small model right and Large model wrong: 9; Small model wrong and Large model right: 14; Small model wrong and Large model wrong: 6. The fourteen in the lower left are what you are paying the large model for, and your escalation rules have to catch them. The nine in the top right are the group people forget: large models are better on average rather than uniformly, and they overthink simple cases.\n\n## The cheapest escalation of all\n\nThere is a version of the third group that costs nothing to catch: you. In work where a person is reading the output anyway (drafting, correspondence, review queues) the escalation rule can be that you notice. Run everything on the cheap model, and when an answer looks thin, ask the expensive one the same question.\n\nThat is a cascade with a human as the confidence signal, and on small volumes it beats any automated version, because you are far better calibrated about your own work than a model reporting on its own certainty.\n\n## The version for a person, not a system\n\nYou do not need code for any of this. The habit is the same:\n\n- Default to the cheap, fast model for everything routine: rewriting, summarising, formatting, ordinary questions.\n- Move to the strong model when the answer matters, when the question is hard, or when the cheap one has given you something you doubt.\n- Notice when you last used the expensive model out of habit rather than need.\n\nPeople who work this way spend a fraction of what people who leave the strongest model selected all day spend, get answers faster for most of their work, and do not lose anything. The cheap model is not a compromise for two thirds of what anybody actually does."},"check":{"question":"Comparing a small and a large model on 100 items, a team finds a group where the small model is right and the large one is wrong. What does that group mean for the design?","options":[{"text":"It is measurement noise from sampling, and running both models at temperature zero would eliminate the group.","correct":false,"tempting":"Dismisses a well-documented pattern as randomness; the group persists at low temperature because large models genuinely overthink simple cases."},{"text":"It shows the large model needs a better prompt, since a stronger model should dominate a weaker one on every item.","correct":false,"tempting":"Assumes strict dominance that does not hold, and prompt tuning does not remove a group that arises from the models differing in kind."},{"text":"It indicates the sample of 100 is too small, and a larger comparison would show the large model ahead on every category.","correct":false,"tempting":"Attributes to sample size a difference that grows rather than vanishes with more items."},{"text":"The models fail on different items, so escalating everything would add errors the small one avoids.","correct":true}]},"prev":{"slug":"caching-and-batching","title":"The two discounts almost nobody uses","url":"https://addaly.com/learn/choosing-your-tools/caching-and-batching"},"next":{"slug":"three-billing-shapes","title":"Subscriptions, credits and pay-as-you-go","url":"https://addaly.com/learn/choosing-your-tools/three-billing-shapes"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Spending money where it matters', in Choosing and Using the Tools. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/choosing-your-tools/routing-by-difficulty (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Spending money where it matters","courseTitle":"Choosing and Using the Tools","module":{"index":4,"slug":"what-it-costs","title":"What it actually costs"},"answer":null,"keyIdea":"A cascade (cheap model everywhere, expensive model only on flagged cases) commonly cuts cost by four fifths at equal or better quality, provided you have measured on a hundred items which cases the cheap model actually gets wrong.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["Large model right","Large model wrong"],"rows":["Small model right","Small model wrong"],"type":"matrix","cells":[["71","9"],["14","6"]],"title":"A hundred items, run through both models","caption":"The fourteen in the lower left are what you are paying the large model for, and your escalation rules have to catch them. The nine in the top right are the group people forget: large models are better on average rather than uniformly, and they overthink simple cases.","cellLabels":[["the cascade is safe here","small right, large wrong"],["escalation has to catch these","the prompt or the task, not the model"]]},"alt":"Figure: A hundred items, run through both models. A two by two grid, rows Small model right and Small model wrong, columns Large model right and Large model wrong: Small model right and Large model right: 71; Small model right and Large model wrong: 9; Small model wrong and Large model right: 14; Small model wrong and Large model wrong: 6. The fourteen in the lower left are what you are paying the large model for, and your escalation rules have to catch them. The nine in the top right are the group people forget: large models are better on average rather than uniformly, and they overthink simple cases."}},"blocks":[{"t":"h2","id":"the-shape-production-systems-actually-have","runs":[{"text":"The shape production systems actually have"}]},{"t":"p","runs":[{"text":"People assume the systems they admire run the best available model on every request. Very few do. The common design is a "},{"text":"cascade","b":true},{"text":": a cheap model handles everything, and only the cases it cannot settle go to an expensive one."}]},{"t":"code","lang":"","label":"Text","text":"1,000 items\n  -> small model handles all 1,000            $0.40\n  -> 120 flagged uncertain go to large model  $2.60\n                                              -----\n                                              $3.00\n  versus all 1,000 on the large model:       $21.60"},{"t":"p","runs":[{"text":"Eighty-six per cent saved, with the hard cases still getting the expensive treatment. This is not a compromise design. On many tasks it also scores "},{"text":"better","i":true},{"text":" than the large model alone, because the escalation step forces a second look at exactly the items most likely to be wrong."}]},{"t":"h2","id":"deciding-what-escalates","runs":[{"text":"Deciding what escalates"}]},{"t":"p","runs":[{"text":"You need a signal for “this one is difficult”, and there are four that work without any special infrastructure."}]},{"t":"p","runs":[{"text":"Ask for a confidence word.","b":true},{"text":" Have the small model end with "},{"text":"CONFIDENT","code":true},{"text":" or "},{"text":"UNSURE","code":true},{"text":". It is not calibrated and it is not a probability, and it still correlates usefully with correctness: well enough to route on. Check the correlation on fifty items before trusting it."}]},{"t":"p","runs":[{"text":"Rules you already know.","b":true},{"text":" Some cases are structurally hard: a document over thirty pages, a ticket from a top-tier customer, an invoice above a threshold, a message in a language the small model handles poorly. Route those up without asking anything."}]},{"t":"p","runs":[{"text":"Disagreement.","b":true},{"text":" Run the small model twice at a non-zero temperature. If the two answers differ, escalate. Cheap, crude, and effective on classification."}]},{"t":"p","runs":[{"text":"Cost of being wrong.","b":true},{"text":" The most important one, and it is a business decision rather than a technical one. Anything where a mistake costs money, safety or a customer goes to the expensive model regardless of how confident anything is."}]},{"t":"h2","id":"the-measurement-that-makes-it-honest","runs":[{"text":"The measurement that makes it honest"}]},{"t":"p","runs":[{"text":"A cascade is only correct if the small model is adequate on the easy cases, and the only way to know is to check."}]},{"t":"p","runs":[{"text":"Take 100 items. Run both models. Compare. You will get four groups, and the interesting one is the fourth:"}]},{"t":"ul","items":[[{"text":"Both right: the cascade is safe here."}],[{"text":"Both wrong: the model is not the problem; the prompt or the task is."}],[{"text":"Large right, small wrong: this is what you are paying the large model for. How many, and do your escalation rules catch them?"}],[{"text":"Small right, large wrong.","b":true},{"text":" This group is always bigger than people expect. Large models are not uniformly better; they are better on average, and they overthink simple cases."}]]},{"t":"p","runs":[{"text":"If your escalation rules catch most of the third group, the cascade is working. If they do not, tighten the rules rather than abandoning the design."}]},{"t":"h2","id":"where-cascades-are-the-wrong-idea","runs":[{"text":"Where cascades are the wrong idea"}]},{"t":"p","runs":[{"text":"When latency matters more than money.","b":true},{"text":" Two model calls take twice as long. For anything a person is waiting on, that can be the wrong trade."}]},{"t":"p","runs":[{"text":"When the volume is small.","b":true},{"text":" A hundred items a day at a cent each is a dollar. Engineering a cascade to save 80% of a dollar is not a good use of an afternoon."}]},{"t":"p","runs":[{"text":"When you cannot measure.","b":true},{"text":" A cascade without the 100-item comparison is a guess that quietly ships worse work. If you are not going to check, use the model you trust."}]},{"t":"h2","id":"the-cheapest-escalation-of-all","runs":[{"text":"The cheapest escalation of all"}]},{"t":"p","runs":[{"text":"There is a version of the third group that costs nothing to catch: you. In work where a person is reading the output anyway (drafting, correspondence, review queues) the escalation rule can be that you notice. Run everything on the cheap model, and when an answer looks thin, ask the expensive one the same question."}]},{"t":"p","runs":[{"text":"That is a cascade with a human as the confidence signal, and on small volumes it beats any automated version, because you are far better calibrated about your own work than a model reporting on its own certainty."}]},{"t":"h2","id":"the-version-for-a-person-not-a-system","runs":[{"text":"The version for a person, not a system"}]},{"t":"p","runs":[{"text":"You do not need code for any of this. The habit is the same:"}]},{"t":"ul","items":[[{"text":"Default to the cheap, fast model for everything routine: rewriting, summarising, formatting, ordinary questions."}],[{"text":"Move to the strong model when the answer matters, when the question is hard, or when the cheap one has given you something you doubt."}],[{"text":"Notice when you last used the expensive model out of habit rather than need."}]]},{"t":"p","runs":[{"text":"People who work this way spend a fraction of what people who leave the strongest model selected all day spend, get answers faster for most of their work, and do not lose anything. The cheap model is not a compromise for two thirds of what anybody actually does."}]}],"check":{"question":[{"text":"Comparing a small and a large model on 100 items, a team finds a group where the small model is right and the large one is wrong. What does that group mean for the design?"}],"options":[{"text":[{"text":"It is measurement noise from sampling, and running both models at temperature zero would eliminate the group."}],"correct":false,"tempting":[{"text":"Dismisses a well-documented pattern as randomness; the group persists at low temperature because large models genuinely overthink simple cases."}]},{"text":[{"text":"It shows the large model needs a better prompt, since a stronger model should dominate a weaker one on every item."}],"correct":false,"tempting":[{"text":"Assumes strict dominance that does not hold, and prompt tuning does not remove a group that arises from the models differing in kind."}]},{"text":[{"text":"It indicates the sample of 100 is too small, and a larger comparison would show the large model ahead on every category."}],"correct":false,"tempting":[{"text":"Attributes to sample size a difference that grows rather than vanishes with more items."}]},{"text":[{"text":"The models fail on different items, so escalating everything would add errors the small one avoids."}],"correct":true,"tempting":null}],"order":[0,3,2,1],"why":null},"prev":{"slug":"caching-and-batching","title":"The two discounts almost nobody uses","minutes":8},"next":{"slug":"three-billing-shapes","title":"Subscriptions, credits and pay-as-you-go","minutes":7}}}