Bigger and crushed, or smaller and clean

The question everybody has

You have 12 GB. You can run a 14B at roughly 4 bits, or an 8B at roughly 8 bits, or a 4B at full precision. All three fit. Which is best?

The general finding, repeated across many evaluations, is that more parameters at lower precision wins, down to about 4 bits per weight. A 14B at Q4_K_M beats an 8B at Q8_0 on most tasks, despite the 8B being nearly lossless and the 14B being visibly compressed.

The mechanism is worth understanding rather than memorising. A larger model has more redundancy: many weights encode overlapping information, so rounding any one of them loses less. It also knows and can do more, and the capability gap between size classes is larger than the gap quantisation opens within one.

Where the rule stops

Below about 4 bits the advantage erodes, and by 2 bits it usually reverses at small and medium sizes. Three reasons:

1. Quantisation error grows faster than linearly as bits fall. Going from 8 to 4 bits costs little; from 4 to 3 costs noticeably more; from 3 to 2 costs a great deal more again.
2. Small models have less redundancy to spend. A 7B has fewer spare parameters encoding the same thing, so each damaged weight matters more. Large models tolerate crushing far better, which is why a 70B at 2.4 bits remains useful while a 7B at 2 bits is not.
3. The scale overhead becomes significant. At 2 nominal bits with per-block scales, the real rate is closer to 2.4–2.6, so you are paying a quarter of your budget on metadata.

So the practical ordering, for a fixed memory budget: prefer the largest model that fits at 4 bits or better; if reaching 4 bits requires a model so large that only 2-bit fits, step down a size class instead, unless that class is 70B or above.

What the rule ignores

Three things make the rule wrong for specific people, and they are common enough to name.

Speed. A 14B at Q4 reads about 8.5 GB per token; an 8B at Q4 reads 4.9. On a bandwidth-limited machine that is nearly a factor of two in tokens per second, and on a laptop the 14B may be unusably slow even though it fits. The rule optimises quality per gigabyte, not quality per second.

Context. A bigger model leaves less room for the KV cache. If your work needs 32k of context, a 14B that fits at 4k and an 8B that fits at 32k are not competing on quality: one of them cannot do the job.

Task specialisation. A 7B trained for code beats a general 14B at code, at half the size. Specialisation is a bigger lever than either parameter count or precision, and it is checked far less often.

A worked comparison

Suppose 12 GB of VRAM, general assistant work, 8k context, about 1 GB overhead. That leaves roughly 10 GB for weights plus cache.

• 14B at Q4_K_M: 8.4 GB weights, about 0.8 GB cache. Fits. Best raw capability of the three.
• 8B at Q6_K: 6.6 GB weights, 1.0 GB cache. Fits easily, roughly 30% faster to generate, quality below the 14B on reasoning and instruction following.
• 8B at Q4_K_M with 32k context: 4.9 GB weights, 4.0 GB cache. Fits, and is the right answer if your documents are long.

There is no universal winner in that list, which is the honest conclusion. The rule chooses the first; your workload may choose the second or third, and it should.

How to settle it in practice

Download two candidates, not five. Run the audition set at temperature 0 against both, and time them. Then ask three questions: is the quality difference larger than a couple of items out of thirty; is the slower one still fast enough to use; and does either fail on context length. Those three answers decide it, and the whole exercise is an afternoon.

The reason to do it rather than follow the rule is that the rule was derived from benchmark averages across many tasks, and you have one task. Averages are a good prior and a poor conclusion.