Choosing a bit width you can defend

The ladder, and what each rung costs

Ordered from least to most compressed, with the honest character of each:

• bf16 / fp16: the reference. Everything else is measured against this.
• fp8 or Q8_0: effectively indistinguishable from the reference on ordinary tasks. Use it when memory allows; there is nothing to think about.
• Q6_K: very close to Q8 at three-quarters of the size. An underused sweet spot for people with a little headroom.
• Q5_K_M: a small, usually unnoticed loss. Sensible when Q4 makes you nervous and Q6 does not fit.
• Q4_K_M: the community default, and correctly so. The knee of the curve for models above about 7B.
• Q3_K_M: noticeably degraded. Instruction adherence weakens first; the model starts ignoring parts of a multi-clause request.
• Q2_K: usually a mistake. A smaller model at Q4 will almost always be better.

The rule that follows

Spend your memory on parameters before you spend it on precision, down to about 4 bits.

Given 8 GB to work with, a 13B at Q4 beats an 8B at Q6 beats a 7B at Q8, on most tasks, most of the time. Given 5 GB, an 8B at Q4 beats a 4B at Q8.

The rule inverts below 4 bits and at small sizes, and both exceptions have the same cause. Quantisation works because large models carry redundancy: many weights contribute a little, so small errors average out. A 1.5B model has less of that redundancy per parameter, so the same bit reduction costs it more. Below about 3B, prefer Q6 or Q8 and accept the smaller model.

What degrades first, and where the benchmarks hide it

The damage is not spread evenly across capabilities, and a general benchmark average conceals this. In rough order of sensitivity:

1. Long-context retrieval. Finding a specific fact deep in a long document depends on fine distinctions between key vectors, which is exactly what coarse quantisation blurs.
2. Non-English performance. Lower-resource languages sit in less well-populated regions of the model’s representation, with less redundancy supporting them. A quantised model can lose noticeably more of its Hindi than its English.
3. Multi-step arithmetic and code. Errors compound; a small increase in per-step error is visible at the end of a chain.
4. Instruction adherence over long outputs. The model complies for two paragraphs and drifts.
5. Short English chat. The last thing to break, and unfortunately the first thing everyone tests.

If any of the first four describe your use, test at your quantisation rather than trusting a published perplexity number.

Perplexity is a weak proxy

Most quantisation comparisons report perplexity on a text corpus. It is cheap, it correlates with quality, and it is not sensitive enough for this decision: two quantisations can be within a hundredth of a perplexity point and behave differently on structured output.

The better measurement, covered in more detail later, is agreement with the full-precision model. Run both on 200 prompts and compare their next-token distributions, or more compare their actual answers on your own examples. A quantisation that agrees with the reference on 96% of your task is a different proposition from one that agrees on 78%, and perplexity may not separate them.

A practical procedure

1. Compute how much memory you have for weights, after subtracting KV cache and about 1 to 2 GB of overhead.
2. Pick the largest model whose Q4_K_M fits.
3. If it fits with room to spare, move up to Q5 or Q6 rather than to a bigger model, unless the next size up also fits at Q4.
4. Run your 40 examples at the chosen quantisation and at one step higher.
5. If the difference is inside the noise, keep the smaller file. If it is not, you have learned something specific to your task that no general table would have told you.

Step 4 costs an hour and settles arguments that otherwise recur for months.

On downloading: repositories from prolific quantisers such as bartowski and the official organisation uploads generally publish the whole ladder with the importance-matrix variants. Take Q4_K_M first, and take one neighbour so you can run step 4 without a second download.