Choosing a model, and designing so you can change your mind

The tiers, and the price gap between them

Almost every provider now ships three sizes, and the naming is cosmetic. What matters is the ratio.

• Small / fast: roughly $0.10 to $0.50 per million input tokens. Classification, routing, extraction from clean text, rewriting, moderation pre-filters.
• Mid: roughly $1 to $5 per million input. Most product work: answering from retrieved documents, summarising, drafting, tool use with a handful of tools.
• Frontier / reasoning: roughly $5 to $30 per million input, and often 3 to 5× that on output. Hard multi-step reasoning, difficult code, long agent chains, ambiguous judgement.

The gap between the small tier and the frontier tier is commonly 30 to 100 times on price and 3 to 10 times on latency. That spread is the single largest lever you have over the economics of your product, and most teams never pull it because they picked the best model on day one and never revisited.

Benchmarks will not tell you which to use

Published scores measure performance on public test sets. Your task is not on any of them. Three specific reasons the ranking does not transfer:

Contamination. Public benchmarks leak into training corpora. A score partly measures memorisation of the test.

Distribution mismatch. Your inputs are customer messages with typos in three languages, or scanned invoices from one supplier, or your own product’s jargon. Nothing in a reasoning benchmark resembles that.

The metric is not your metric. A benchmark rewards the right final answer. Your feature might need to refuse when unsure, cite a source, keep to a format, and finish in 800 milliseconds. A model can win on the first and lose badly on the rest.

The replacement takes an afternoon: twenty of your own cases with known correct answers, run against three candidate models, cost and latency recorded alongside quality. Twenty is small enough to build today and large enough to eliminate most candidates. It will disagree with the leaderboard often enough to pay for itself immediately.

Route, do not choose

The better design is not one model. It is a cheap model first, escalating when needed.

def answer(q, docs):
    draft = small(q, docs)          # 40x cheaper
    if confident(draft):            # a check you define, not the model's opinion
        return draft
    return frontier(q, docs)

confident() is the interesting part, and it must not be the model’s self-report. Workable signals: the extraction validated against a schema; the retrieved documents scored above a similarity floor; the answer quoted a source you can find in the text; the input was under a length where the small model is known to hold up. On typical support and extraction workloads, 70 to 90 per cent of traffic never needs the expensive model, which turns a 40× price gap into a 5× effective one.

The same logic applies in reverse for latency. A small model answering in 400 milliseconds and being right 92% of the time can beat a frontier model that is right 96% of the time and takes four seconds, if the failure is cheap and recoverable.

Open weights are a real option

Open-weight families (Llama, Qwen, Mistral, Gemma, DeepSeek, Phi) are behind the frontier on hard reasoning and roughly level on the ordinary tasks that make up most product work: classification, extraction, summarising, rewriting, simple tool use. You can run them three ways:

• Hosted by someone else, at prices commonly 5 to 20× below closed frontier models. The API is usually OpenAI-compatible, so it is a base-URL change.
• On your own machine for free, via Ollama or llama.cpp, the next lesson.
• On your own server with vLLM, which is the option that makes sense when you have steady high volume, and only then.

The reason to care even if you never use them: an open-weight fallback is the only fallback nobody can take away from you. Pricing changes, models get retired, regions get blocked, accounts get suspended. A path that runs on hardware you control is insurance.

Design for replacement

The best model on any given task will change several times a year, and code written against one vendor’s exact response shape will not follow. Keep one narrow interface:

class LLM(Protocol):
    def complete(self, system: str, messages: list, *,
                 tools=None, max_tokens=1024,
                 temperature=0.0) -> Completion: ...

One adapter per provider behind it. Everything above it (prompts, retrieval, validation, evaluation) stays untouched when you swap. Gateways such as OpenRouter give you the same effect over HTTP with a single key and cross-vendor fallback, at the cost of another party in the path and slightly less access to provider-specific features like prompt caching.

The honest summary: you will change models. Make that a config edit and a re-run of your twenty cases, not a refactor.