Measuring the damage, properly

The number everybody quotes cannot detect the problem

Perplexity is the exponentiated average negative log-likelihood of a test corpus: how surprised the model is by text it should find ordinary. Every quantisation comparison publishes it, usually as a table of deltas: fp16 at 6.21, Q4_K_M at 6.24.

The trouble is what that average hides. Perplexity is dominated by the vast majority of tokens that are easy and that every version of the model gets right. The tokens where quantisation actually hurts (the closing brace, the correct API name, the eleventh constraint in a specification, the rare word in Tamil) are a tiny fraction of the corpus and move the average by almost nothing.

So a 0.5% perplexity delta is compatible with a model that has become clearly worse at code and clearly worse in one of its languages. Perplexity is not useless; it will catch a badly broken quant. It cannot rank two reasonable ones, and it is quoted as though it can.

A better cheap signal: divergence from the original

Instead of asking how surprised the quantised model is by a corpus, ask how differently it distributes probability compared with the model it came from. Feed the same tokens to both, and compare the full output distributions with KL divergence.

# generate a reference distribution file with the full-precision model
llama-perplexity -m model-f16.gguf -f test.txt --kl-divergence-base ref.dat

# then score the quant against it
llama-perplexity -m model-Q4_K_M.gguf -f test.txt --kl-divergence ref.dat

The useful outputs are the mean KL divergence and, more importantly, the top-token agreement rate and the 99th-percentile divergence. Mean divergence tells you the typical difference; the tail tells you how often the quant disagrees badly, which is the thing that produces a visibly wrong answer. Two quants with the same mean and different tails behave very differently in practice.

This is still a proxy. It measures difference from the original, not correctness, and a quant that diverges in a harmless direction scores the same as one that diverges in a harmful one.

The measurement that answers your actual question

Your thirty-item audition set, run at temperature 0 against both the full-precision model and each candidate quant, graded blind.

Temperature 0 matters here: at any positive temperature you are measuring sampling noise as well as quantisation, and you will need many more items to see through it. Greedy decoding removes that variable entirely.

Design the set so it can detect what quantisation actually damages. Include:

• Exact-syntax items. Valid JSON against a schema, a regex, a shell command, a piece of code that must compile. Score these automatically (parse the JSON, run the code) because they are objective and cheap.
• Long-instruction items. A prompt with twelve numbered constraints. Score by counting how many were honoured. This is the most sensitive test of quantisation damage there is and almost nobody runs it.
• Long-context items. A fact placed 8,000 tokens in, then asked about. Sensitive to KV-cache quantisation specifically.
• Items in every language you use. Quantisation damages low-resource languages first, and an English-only test will report that nothing happened.
• Arithmetic and rare-name items, where a single wrong token is unambiguously wrong.

Reading a result honestly

Thirty items is a small sample. A difference of one or two items between two quants is noise; a difference of eight is real. If the two candidates come out within a couple of items of each other, take the smaller or faster one and stop measuring: you have established that the difference is below your ability to detect, which is a legitimate and useful finding.

Run the same set against the unquantised model too, even if you will never deploy it. Without that reference you cannot tell a quantisation failure from a model that cannot do the task, and those two conclusions lead to completely different next steps.

What to write down

Model, quant, engine version, context length, sampler settings, the date, the scores. Engines change, defaults change, and a quant that was fine in one version can regress in the next. Six lines in a text file, and the next time somebody claims a quant is bad you can answer with a measurement rather than an impression.