Proving what a quantisation cost

Four methods, from weakest to strongest

You have a Q4 file and a claim that it is nearly lossless. Here is how to check, in ascending order of usefulness.

1. Perplexity

./llama-perplexity -m model-f16.gguf -f wiki.test.raw -c 4096
./llama-perplexity -m model-Q4_K_M.gguf -f wiki.test.raw -c 4096

Perplexity is the exponentiated average negative log-likelihood on a fixed corpus: how surprised the model is by text it did not write. Lower is better, and the comparison is only meaningful between two models sharing a tokenizer, which is exactly the quantisation case.

A rise from 6.10 to 6.18 is small. A rise to 7.4 is not. But two quantisations can sit within a hundredth of each other and behave differently on structured output, so perplexity is a screening test, not a verdict.

2. Divergence from the reference

Much more sensitive, and the right default. Instead of asking how surprised each model is by a corpus, ask how far the quantised model’s output distribution has moved from the full-precision model’s.

# produce a reference from the unquantised model
./llama-perplexity -m model-f16.gguf -f wiki.test.raw --kl-divergence-base ref.dat

# compare a quantisation against it
./llama-perplexity -m model-Q4_K_M.gguf -f wiki.test.raw --kl-divergence ref.dat

The output includes the mean Kullback-Leibler divergence and, more usefully, the fraction of tokens where the top choice differs between the two models. That last figure is directly interpretable: if the quantised model picks a different most-likely token 1.5% of the time, you know what you bought. Above about 5%, expect visible behavioural differences.

Report the tail as well as the mean. A quantisation with a low mean divergence and a heavy 99th percentile is one that is usually identical and occasionally very different, which is worse for a product than a uniform small degradation.

3. Your own task

Run your forty held-out examples through both files at temperature 0, and compare the answers directly. Two numbers matter: the task score for each, and the agreement rate between them.

Agreement is the sharper instrument on small sets. With forty examples, a two-point score difference is noise; but if the two models produce different answers on nine of forty items, something real has changed even when the scores happen to match.

4. The specific capabilities quantisation damages

From the earlier lesson: long-context retrieval, lower-resource languages, multi-step arithmetic and long-output instruction adherence degrade before short English chat does. So test those deliberately rather than hoping a general set covers them.

• Ten questions whose answers sit deep in a long document, at your real context length.
• Ten items in each non-English language you serve.
• Ten items requiring a strict output format held over a long response.

This is thirty items and it is where the difference will appear if there is one.

Test the cache separately

KV cache quantisation is a different decision from weight quantisation and is frequently changed at the same time, which confounds both. Run four configurations if the setting matters to you: full weights with full cache, quantised weights with full cache, full weights with quantised cache, and both. The long-context items are where cache quantisation shows up, and it is often the cause when a model that tested well behaves oddly on long documents.

What to write down

For any quantisation you ship:

model      Qwen3-8B @ 9a3f21c
quant      Q4_K_M (imatrix, bartowski, rev 2f8c1)
perplexity 6.10 -> 6.17 on wiki.test.raw, c=4096
top-token  differs on 1.4% of tokens vs f16
task       0.86 -> 0.85 on held-out v4 (40 items), agreement 37/40
long-ctx   0.90 -> 0.80 on the 10 deep-retrieval items
decision   accepted; long-context path routed to Q6

Six lines, and it turns a choice somebody made once into a documented decision anybody can revisit.