{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"choosing-a-model","title":"Choosing a model, and designing so you can change your mind","url":"https://addaly.com/learn/building-with-ai/choosing-a-model","position":8,"of":80,"minutes":9,"answer":"The price gap between model tiers is 30-100x, benchmarks do not transfer to your task, so route cheap-first against twenty of your own cases and keep the provider behind one narrow interface.","keyIdea":"The price gap between model tiers is 30-100x, benchmarks do not transfer to your task, so route cheap-first against twenty of your own cases and keep the provider behind one narrow interface."},"body":{"format":"markdown","text":"## The tiers, and the price gap between them\n\nAlmost every provider now ships three sizes, and the naming is cosmetic. What matters is the ratio.\n\n- **Small / fast**: roughly $0.10 to $0.50 per million input tokens. Classification, routing, extraction from clean text, rewriting, moderation pre-filters.\n- **Mid**: roughly $1 to $5 per million input. Most product work: answering from retrieved documents, summarising, drafting, tool use with a handful of tools.\n- **Frontier / reasoning**: roughly $5 to $30 per million input, and often 3 to 5× that on output. Hard multi-step reasoning, difficult code, long agent chains, ambiguous judgement.\n\nThe gap between the small tier and the frontier tier is commonly **30 to 100 times** on price and 3 to 10 times on latency. That spread is the single largest lever you have over the economics of your product, and most teams never pull it because they picked the best model on day one and never revisited.\n\n## Benchmarks will not tell you which to use\n\nPublished scores measure performance on public test sets. Your task is not on any of them. Three specific reasons the ranking does not transfer:\n\n**Contamination.** Public benchmarks leak into training corpora. A score partly measures memorisation of the test.\n\n**Distribution mismatch.** Your inputs are customer messages with typos in three languages, or scanned invoices from one supplier, or your own product's jargon. Nothing in a reasoning benchmark resembles that.\n\n**The metric is not your metric.** A benchmark rewards the right final answer. Your feature might need to refuse when unsure, cite a source, keep to a format, and finish in 800 milliseconds. A model can win on the first and lose badly on the rest.\n\nThe replacement takes an afternoon: **twenty of your own cases with known correct answers**, run against three candidate models, cost and latency recorded alongside quality. Twenty is small enough to build today and large enough to eliminate most candidates. It will disagree with the leaderboard often enough to pay for itself immediately.\n\n## Route, do not choose\n\nThe better design is not one model. It is a cheap model first, escalating when needed.\n\n```python\ndef answer(q, docs):\n    draft = small(q, docs)          # 40x cheaper\n    if confident(draft):            # a check you define, not the model's opinion\n        return draft\n    return frontier(q, docs)\n```\n\n`confident()` is the interesting part, and it must not be the model's self-report. Workable signals: the extraction validated against a schema; the retrieved documents scored above a similarity floor; the answer quoted a source you can find in the text; the input was under a length where the small model is known to hold up. On typical support and extraction workloads, 70 to 90 per cent of traffic never needs the expensive model, which turns a 40× price gap into a 5× effective one.\n\nThe same logic applies in reverse for latency. A small model answering in 400 milliseconds and being right 92% of the time can beat a frontier model that is right 96% of the time and takes four seconds, if the failure is cheap and recoverable.\n\n## Open weights are a real option\n\nOpen-weight families (Llama, Qwen, Mistral, Gemma, DeepSeek, Phi) are behind the frontier on hard reasoning and roughly level on the ordinary tasks that make up most product work: classification, extraction, summarising, rewriting, simple tool use. You can run them three ways:\n\n- **Hosted by someone else**, at prices commonly 5 to 20× below closed frontier models. The API is usually OpenAI-compatible, so it is a base-URL change.\n- **On your own machine** for free, via Ollama or llama.cpp, the next lesson.\n- **On your own server** with vLLM, which is the option that makes sense when you have steady high volume, and only then.\n\nThe reason to care even if you never use them: an open-weight fallback is the only fallback nobody can take away from you. Pricing changes, models get retired, regions get blocked, accounts get suspended. A path that runs on hardware you control is insurance.\n\n## Design for replacement\n\nThe best model on any given task will change several times a year, and code written against one vendor's exact response shape will not follow. Keep one narrow interface:\n\n```python\nclass LLM(Protocol):\n    def complete(self, system: str, messages: list, *,\n                 tools=None, max_tokens=1024,\n                 temperature=0.0) -> Completion: ...\n```\n\nOne adapter per provider behind it. Everything above it (prompts, retrieval, validation, evaluation) stays untouched when you swap. Gateways such as OpenRouter give you the same effect over HTTP with a single key and cross-vendor fallback, at the cost of another party in the path and slightly less access to provider-specific features like prompt caching.\n\n> Figure: A forty-times price gap, spent three ways. Bars (dollars per 1,000 requests): Small model on everything 1; Cheap first, 20% escalates 12.8; Frontier model on everything 60. One thousand requests of 2,000 input and 400 output tokens. Small tier at $0.25 and $1.25 per million; frontier at $15 and $75. Routing eighty per cent of traffic to the small model turns a 60-fold gap into a 5-fold one, and the router costs a fraction of a cent.\n\nThe honest summary: **you will change models. Make that a config edit and a re-run of your twenty cases, not a refactor.**"},"check":{"question":"A team wants to cut model spend without losing quality on a support-reply feature. Which change has the best chance of a large saving?","options":[{"text":"Send every request to the small model and accept a slightly lower quality bar.","correct":false,"tempting":"Trades quality uniformly, including on the hard minority of cases where the small model fails badly and the cost of being wrong is highest."},{"text":"Answer with the small model first and escalate only when a check you control (schema validity, retrieval score, a verifiable quote) says the draft is untrustworthy.","correct":true},{"text":"Reduce max_tokens so answers are shorter.","correct":false,"tempting":"Saves on the output side only, and risks truncation; on retrieval-heavy features input tokens usually dominate the bill."},{"text":"Ask the model to rate its own confidence and re-ask the frontier model when the rating is low.","correct":false,"tempting":"Uses a self-report as a gate, but stated confidence is generated the same way as the rest of the text and is not calibrated to correctness."}]},"prev":{"slug":"retries-that-do-not-make-it-worse","title":"Errors, retries, and how a retry loop takes down your own service","url":"https://addaly.com/learn/building-with-ai/retries-that-do-not-make-it-worse"},"next":{"slug":"a-model-on-your-own-machine","title":"Running a model on a laptop with no GPU","url":"https://addaly.com/learn/building-with-ai/a-model-on-your-own-machine"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Choosing a model, and designing so you can change your mind', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/choosing-a-model (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Choosing a model, and designing so you can change your mind","courseTitle":"Building with AI","module":{"index":1,"slug":"the-call-underneath-everything","title":"The call underneath everything"},"answer":null,"keyIdea":"The price gap between model tiers is 30-100x, benchmarks do not transfer to your task, so route cheap-first against twenty of your own cases and keep the provider behind one narrow interface.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"dollars per 1,000 requests","title":"A forty-times price gap, spent three ways","series":[{"label":"Small model on everything","value":1},{"label":"Cheap first, 20% escalates","value":12.8,"highlight":true},{"label":"Frontier model on everything","value":60}],"caption":"One thousand requests of 2,000 input and 400 output tokens. Small tier at $0.25 and $1.25 per million; frontier at $15 and $75. Routing eighty per cent of traffic to the small model turns a 60-fold gap into a 5-fold one, and the router costs a fraction of a cent."},"alt":"Figure: A forty-times price gap, spent three ways. Bars (dollars per 1,000 requests): Small model on everything 1; Cheap first, 20% escalates 12.8; Frontier model on everything 60. One thousand requests of 2,000 input and 400 output tokens. Small tier at $0.25 and $1.25 per million; frontier at $15 and $75. Routing eighty per cent of traffic to the small model turns a 60-fold gap into a 5-fold one, and the router costs a fraction of a cent."}},"blocks":[{"t":"h2","id":"the-tiers-and-the-price-gap-between-them","runs":[{"text":"The tiers, and the price gap between them"}]},{"t":"p","runs":[{"text":"Almost every provider now ships three sizes, and the naming is cosmetic. What matters is the ratio."}]},{"t":"ul","items":[[{"text":"Small / fast","b":true},{"text":": roughly $0.10 to $0.50 per million input tokens. Classification, routing, extraction from clean text, rewriting, moderation pre-filters."}],[{"text":"Mid","b":true},{"text":": roughly $1 to $5 per million input. Most product work: answering from retrieved documents, summarising, drafting, tool use with a handful of tools."}],[{"text":"Frontier / reasoning","b":true},{"text":": roughly $5 to $30 per million input, and often 3 to 5× that on output. Hard multi-step reasoning, difficult code, long agent chains, ambiguous judgement."}]]},{"t":"p","runs":[{"text":"The gap between the small tier and the frontier tier is commonly "},{"text":"30 to 100 times","b":true},{"text":" on price and 3 to 10 times on latency. That spread is the single largest lever you have over the economics of your product, and most teams never pull it because they picked the best model on day one and never revisited."}]},{"t":"h2","id":"benchmarks-will-not-tell-you-which-to-use","runs":[{"text":"Benchmarks will not tell you which to use"}]},{"t":"p","runs":[{"text":"Published scores measure performance on public test sets. Your task is not on any of them. Three specific reasons the ranking does not transfer:"}]},{"t":"p","runs":[{"text":"Contamination.","b":true},{"text":" Public benchmarks leak into training corpora. A score partly measures memorisation of the test."}]},{"t":"p","runs":[{"text":"Distribution mismatch.","b":true},{"text":" Your inputs are customer messages with typos in three languages, or scanned invoices from one supplier, or your own product’s jargon. Nothing in a reasoning benchmark resembles that."}]},{"t":"p","runs":[{"text":"The metric is not your metric.","b":true},{"text":" A benchmark rewards the right final answer. Your feature might need to refuse when unsure, cite a source, keep to a format, and finish in 800 milliseconds. A model can win on the first and lose badly on the rest."}]},{"t":"p","runs":[{"text":"The replacement takes an afternoon: "},{"text":"twenty of your own cases with known correct answers","b":true},{"text":", run against three candidate models, cost and latency recorded alongside quality. Twenty is small enough to build today and large enough to eliminate most candidates. It will disagree with the leaderboard often enough to pay for itself immediately."}]},{"t":"h2","id":"route-do-not-choose","runs":[{"text":"Route, do not choose"}]},{"t":"p","runs":[{"text":"The better design is not one model. It is a cheap model first, escalating when needed."}]},{"t":"code","lang":"python","label":"Python","text":"def answer(q, docs):\n    draft = small(q, docs)          # 40x cheaper\n    if confident(draft):            # a check you define, not the model's opinion\n        return draft\n    return frontier(q, docs)"},{"t":"p","runs":[{"text":"confident()","code":true},{"text":" is the interesting part, and it must not be the model’s self-report. Workable signals: the extraction validated against a schema; the retrieved documents scored above a similarity floor; the answer quoted a source you can find in the text; the input was under a length where the small model is known to hold up. On typical support and extraction workloads, 70 to 90 per cent of traffic never needs the expensive model, which turns a 40× price gap into a 5× effective one."}]},{"t":"p","runs":[{"text":"The same logic applies in reverse for latency. A small model answering in 400 milliseconds and being right 92% of the time can beat a frontier model that is right 96% of the time and takes four seconds, if the failure is cheap and recoverable."}]},{"t":"h2","id":"open-weights-are-a-real-option","runs":[{"text":"Open weights are a real option"}]},{"t":"p","runs":[{"text":"Open-weight families (Llama, Qwen, Mistral, Gemma, DeepSeek, Phi) are behind the frontier on hard reasoning and roughly level on the ordinary tasks that make up most product work: classification, extraction, summarising, rewriting, simple tool use. You can run them three ways:"}]},{"t":"ul","items":[[{"text":"Hosted by someone else","b":true},{"text":", at prices commonly 5 to 20× below closed frontier models. The API is usually OpenAI-compatible, so it is a base-URL change."}],[{"text":"On your own machine","b":true},{"text":" for free, via Ollama or llama.cpp, the next lesson."}],[{"text":"On your own server","b":true},{"text":" with vLLM, which is the option that makes sense when you have steady high volume, and only then."}]]},{"t":"p","runs":[{"text":"The reason to care even if you never use them: an open-weight fallback is the only fallback nobody can take away from you. Pricing changes, models get retired, regions get blocked, accounts get suspended. A path that runs on hardware you control is insurance."}]},{"t":"h2","id":"design-for-replacement","runs":[{"text":"Design for replacement"}]},{"t":"p","runs":[{"text":"The best model on any given task will change several times a year, and code written against one vendor’s exact response shape will not follow. Keep one narrow interface:"}]},{"t":"code","lang":"python","label":"Python","text":"class LLM(Protocol):\n    def complete(self, system: str, messages: list, *,\n                 tools=None, max_tokens=1024,\n                 temperature=0.0) -> Completion: ..."},{"t":"p","runs":[{"text":"One adapter per provider behind it. Everything above it (prompts, retrieval, validation, evaluation) stays untouched when you swap. Gateways such as OpenRouter give you the same effect over HTTP with a single key and cross-vendor fallback, at the cost of another party in the path and slightly less access to provider-specific features like prompt caching."}]},{"t":"p","runs":[{"text":"The honest summary: "},{"text":"you will change models. Make that a config edit and a re-run of your twenty cases, not a refactor.","b":true}]}],"check":{"question":[{"text":"A team wants to cut model spend without losing quality on a support-reply feature. Which change has the best chance of a large saving?"}],"options":[{"text":[{"text":"Send every request to the small model and accept a slightly lower quality bar."}],"correct":false,"tempting":[{"text":"Trades quality uniformly, including on the hard minority of cases where the small model fails badly and the cost of being wrong is highest."}]},{"text":[{"text":"Answer with the small model first and escalate only when a check you control (schema validity, retrieval score, a verifiable quote) says the draft is untrustworthy."}],"correct":true,"tempting":null},{"text":[{"text":"Reduce max_tokens so answers are shorter."}],"correct":false,"tempting":[{"text":"Saves on the output side only, and risks truncation; on retrieval-heavy features input tokens usually dominate the bill."}]},{"text":[{"text":"Ask the model to rate its own confidence and re-ask the frontier model when the rating is low."}],"correct":false,"tempting":[{"text":"Uses a self-report as a gate, but stated confidence is generated the same way as the rest of the text and is not calibrated to correctness."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"retries-that-do-not-make-it-worse","title":"Errors, retries, and how a retry loop takes down your own service","minutes":9},"next":{"slug":"a-model-on-your-own-machine","title":"Running a model on a laptop with no GPU","minutes":10}}}