Small, medium, large: what the size label buys

One family, three very different tools

Every serious model family ships in at least three sizes, and the gap between them is larger than the gap between competing families at the same size. Choosing the size correctly saves more money than choosing the brand correctly, and it is the decision most people never make at all, because the chat app picks for them.

The rough shape, holding for several years now:

• The small one: mini, nano, flash, lite, haiku, or an open model of 3 to 9 billion parameters. Answers in under a second. Costs somewhere between one twentieth and one thirtieth of the large one per token.
• The middle one: sonnet, pro, medium, or an open model of 20 to 40 billion. Handles most real work.
• The large one: opus, ultra, pro at the top of some families, or an open model of 70 billion and up. Slower, several times the price, better at exactly the tasks below.

Where the money actually goes

Here is the arithmetic that makes this concrete. You have 50,000 support tickets to sort into eight categories. Each ticket plus instructions is about 600 tokens in, 20 tokens out.

input tokens  = 50,000 x 600 = 30,000,000  (30M)
output tokens = 50,000 x  20 =  1,000,000  ( 1M)

small model  @ $0.15 /M in, $0.60 /M out  ->  $4.50 + $0.60 =  $5.10
large model  @ $3.00 /M in, $15.00 /M out ->  $90.00 + $15.00 = $105.00

Twenty times the price for a job where, if you test it honestly, the small model will very likely match the large one, because classification into eight known buckets is not a hard task. Those exact prices will be out of date by the time you read this. The ratio between them has been stable for years, and the ratio is what the decision turns on.

What the small one is good at

Not “simple things”. A more useful line: tasks where the answer is already present in the input and the model only has to transform it.

• Classifying into categories you define
• Extracting fields from a document you supply
• Rewriting, shortening, changing tone
• Translating between well-resourced languages
• Formatting, tidying, converting between structures
• Answering questions about a text you pasted

Where it falls over, and why

Knowledge it has to supply itself. A small model has fewer parameters and therefore stores less of the world. Ask a 7B model about a mid-sized company’s history or a specific clause of your country’s tax code and it will produce a fluent, confidently wrong answer. It does not know that it does not know. This is the single most common way small models embarrass people.

Long instructions with several conditions. Give a small model a prompt with seven rules and it will follow five. The failure is not random: the middle rules are the ones dropped.

Multi-step reasoning. Anything where step three depends on getting step two right. Errors compound and the small model has no habit of checking itself.

Long inputs. Even when the window is large, quality on a 200-page document degrades much faster for a small model.

The ladder, as a working method

Do not pick a size by reputation. Pick it by starting at the bottom.

1. Run the task on the smallest model in the family.
2. Look at twenty outputs yourself, not two.
3. If they are right, stop. You are done, and you are paying a twentieth of what you might have.
4. If they are wrong, look at how they are wrong before moving up. Wrong facts means move up. Ignored instructions often means rewrite the prompt instead, and the same small model then passes.

That last point saves a lot of money. A great many “we need the big model” conclusions are “our prompt had seven rules in a paragraph” and the fix was a numbered list.

The mixed setup

The version of this that professionals actually run: small model for the bulk pass, large model for the cases the small one flags as uncertain or that fall into a high-cost category. Ninety per cent of the volume goes cheap, ten per cent goes expensive, and the average price lands near the small model while the quality lands near the large one.

You can do this without any special software. Run the small model, have it output a confidence word alongside its answer, and route the low-confidence ones to a second pass by hand or by a second call. It is twenty lines of code, and it is the standard shape of production systems that people assume are running the frontier model everywhere.