Choosing a Model for the Job

Write the constraints before you look at candidates

Five numbers, on paper, before any browsing:

1. Memory budget on the machine that will actually run it.
2. Latency budget at p95, from a real user’s point of view.
3. Requests per day.
4. Licence constraint (commercial, redistributable, attribution acceptable).
5. Whether data may leave the building.

Most shortlists collapse to two or three candidates immediately. That is the point.

The memory arithmetic

Weights = parameters × bytes per weight.

• bf16 ≈ 2 bytes → an 8B model is about 16 GB
• Q8 ≈ 1 byte → about 8.5 GB
• Q4_K_M ≈ 0.6 bytes → about 4.9 GB

KV cache = 2 × layers × kv_heads × head_dim × bytes, per token. For Llama-3.1-8B (32 layers, 8 KV heads, head dim 128, fp16): 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KB per token.

• 8k context → about 1.0 GB
• 32k context → about 4.2 GB

That arithmetic is why a model that loads happily at 2k context dies at 32k. Add roughly 1 GB of runtime overhead, and aim for weights + KV + 1 GB under 90% of your budget.

If you have 8 GB of RAM and no discrete GPU

This is most people, and most writing about local models pretends otherwise. Honestly:

• 3B to 4B instruct at Q4. Comfortable. About 2.5 GB of weights, room for 8k context, roughly 8 to 20 tokens per second on a recent CPU. Genuinely good at extraction, classification, tagging, rewriting, short summaries.
• 7B to 8B at Q4. Fits at about 4.9 GB, but leaves little for context and nothing for your browser. Expect 4 to 10 tokens per second. Fine for overnight batch work, painful for chat.
• 14B and above. Not on that machine. Not “slow”: not usable.

Apple Silicon changes the maths, because unified memory is shared with the GPU and bandwidth is high. A 16 GB M-series laptop behaves roughly like a 14B-at-Q4 machine, which is a real capability difference from a 16 GB x86 laptop with integrated graphics.

On mixture-of-experts: a 30B model with 3B active still needs the whole 30B resident (around 18 GB at Q4) but computes at roughly 3B speed. Excellent if you have the RAM. Irrelevant if you do not.

And renting is not cheating. OpenRouter, Together, Fireworks, DeepInfra and others serve these same open weights per token. The freedom of open weights is that you could run it yourself and nobody can take the model away; it does not oblige you to buy a GPU. Self-host when data cannot leave, when volume makes it cheaper, or when you need it offline.

Match the shape of the task

• Extraction, classification, routing. Smallest instruct model that passes. Use JSON schema or grammar-constrained decoding, temperature 0. Model choice matters less than constraints here.
• Long documents. Advertised context is not usable context. Most models degrade well before their maximum. Test retrieval from the middle of your own longest document before believing the number on the card.
• Code. Use the coder-tuned variant and check which languages the card names. The gap between a general and a coder model of the same size is large.
• Tool use and agents. Check the card explicitly claims function-calling training, then test with your real schemas. This capability varies far more between models than chat quality does.
• Non-English. Check the language list, then test. Also check tokenizer efficiency: some models spend two or three times as many tokens on non-Latin scripts, which costs you money and eats your effective context at the same time.

The shortlist rule

Three candidates. No more than one size class apart. All licence-clear. Then stop reading and start measuring, which is the next lesson.