{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"measuring-quantisation-damage","title":"Measuring the damage, properly","url":"https://addaly.com/learn/running-models-yourself/measuring-quantisation-damage","position":25,"of":82,"minutes":9,"answer":"Perplexity averages away exactly the tokens quantisation damages: measure with KL divergence against the original and with a task set containing exact-syntax, long-instruction and non-English items at temperature 0.","keyIdea":"Perplexity averages away exactly the tokens quantisation damages: measure with KL divergence against the original and with a task set containing exact-syntax, long-instruction and non-English items at temperature 0."},"body":{"format":"markdown","text":"## The number everybody quotes cannot detect the problem\n\nPerplexity is the exponentiated average negative log-likelihood of a test corpus: how surprised the model is by text it should find ordinary. Every quantisation comparison publishes it, usually as a table of deltas: fp16 at 6.21, Q4_K_M at 6.24.\n\nThe trouble is what that average hides. Perplexity is dominated by the vast majority of tokens that are easy and that every version of the model gets right. The tokens where quantisation actually hurts (the closing brace, the correct API name, the eleventh constraint in a specification, the rare word in Tamil) are a tiny fraction of the corpus and move the average by almost nothing.\n\nSo a 0.5% perplexity delta is compatible with a model that has become clearly worse at code and clearly worse in one of its languages. Perplexity is not useless; it will catch a badly broken quant. It cannot rank two reasonable ones, and it is quoted as though it can.\n\n## A better cheap signal: divergence from the original\n\nInstead of asking how surprised the quantised model is by a corpus, ask how differently it distributes probability compared with the model it came from. Feed the same tokens to both, and compare the full output distributions with KL divergence.\n\n```bash\n# generate a reference distribution file with the full-precision model\nllama-perplexity -m model-f16.gguf -f test.txt --kl-divergence-base ref.dat\n\n# then score the quant against it\nllama-perplexity -m model-Q4_K_M.gguf -f test.txt --kl-divergence ref.dat\n```\n\nThe useful outputs are the mean KL divergence and, more importantly, the top-token agreement rate and the 99th-percentile divergence. Mean divergence tells you the typical difference; the tail tells you how often the quant disagrees badly, which is the thing that produces a visibly wrong answer. Two quants with the same mean and different tails behave very differently in practice.\n\n> Figure: A set that can detect the damage, and one that cannot. Items that will show it: JSON scored by a parser, not by eye; A prompt with twelve numbered constraints; A fact placed 8,000 tokens into the input; Items in every language you actually use; Arithmetic and rare API names. Items that will report nothing: Open-ended English chat, graded by feel; Short questions with short answers; A perplexity delta from 6.21 to 6.24; Anything run once at temperature 0.7; Thirty items all from the easy case. Perplexity averages over the vast majority of tokens every version of the model gets right, and the tokens quantisation damages are a tiny fraction of any corpus. Run the left-hand set at temperature 0 against the full-precision weights as well, or a quantisation failure and a model that simply cannot do the task look identical.\n\nThis is still a proxy. It measures difference from the original, not correctness, and a quant that diverges in a harmless direction scores the same as one that diverges in a harmful one.\n\n## The measurement that answers your actual question\n\nYour thirty-item audition set, run at temperature 0 against both the full-precision model and each candidate quant, graded blind.\n\nTemperature 0 matters here: at any positive temperature you are measuring sampling noise as well as quantisation, and you will need many more items to see through it. Greedy decoding removes that variable entirely.\n\nDesign the set so it can detect what quantisation actually damages. Include:\n\n- **Exact-syntax items.** Valid JSON against a schema, a regex, a shell command, a piece of code that must compile. Score these automatically (parse the JSON, run the code) because they are objective and cheap.\n- **Long-instruction items.** A prompt with twelve numbered constraints. Score by counting how many were honoured. This is the most sensitive test of quantisation damage there is and almost nobody runs it.\n- **Long-context items.** A fact placed 8,000 tokens in, then asked about. Sensitive to KV-cache quantisation specifically.\n- **Items in every language you use.** Quantisation damages low-resource languages first, and an English-only test will report that nothing happened.\n- **Arithmetic and rare-name items**, where a single wrong token is unambiguously wrong.\n\n## Reading a result honestly\n\nThirty items is a small sample. A difference of one or two items between two quants is noise; a difference of eight is real. If the two candidates come out within a couple of items of each other, take the smaller or faster one and stop measuring: you have established that the difference is below your ability to detect, which is a legitimate and useful finding.\n\nRun the same set against the *unquantised* model too, even if you will never deploy it. Without that reference you cannot tell a quantisation failure from a model that cannot do the task, and those two conclusions lead to completely different next steps.\n\n## What to write down\n\nModel, quant, engine version, context length, sampler settings, the date, the scores. Engines change, defaults change, and a quant that was fine in one version can regress in the next. Six lines in a text file, and the next time somebody claims a quant is bad you can answer with a measurement rather than an impression."},"check":{"question":"A quant shows a perplexity increase of 0.4% and a mean KL divergence against fp16 that looks small, yet users report it producing invalid JSON far more often. Which measurement would most likely have predicted this?","options":[{"text":"Perplexity measured on a corpus of JSON documents rather than on general prose.","correct":false,"tempting":"Moves the corpus but keeps the averaging that hides rare critical tokens; most JSON tokens are as easy as most prose tokens."},{"text":"A larger evaluation corpus, since 0.4% is within noise at typical perplexity test sizes.","correct":false,"tempting":"Treats a systematic blind spot as a sample-size issue; more of the same measurement reports the same average more precisely."},{"text":"The tail of the divergence distribution and a schema-validated generation test, since the failure is rare disagreements on structurally critical tokens rather than a shift in the typical case.","correct":true},{"text":"Top-token agreement rate on the same corpus, which would have shown the model choosing different tokens overall.","correct":false,"tempting":"Closer, but still an aggregate over ordinary text; agreement can stay high while the few decisive tokens flip."}]},"prev":{"slug":"quantising-a-model-yourself","title":"Quantising a model yourself","url":"https://addaly.com/learn/running-models-yourself/quantising-a-model-yourself"},"next":{"slug":"the-size-precision-frontier","title":"Bigger and crushed, or smaller and clean","url":"https://addaly.com/learn/running-models-yourself/the-size-precision-frontier"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Measuring the damage, properly', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/measuring-quantisation-damage (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Measuring the damage, properly","courseTitle":"Running Models Yourself","module":{"index":3,"slug":"quantisation-and-formats","title":"Quantisation and model formats"},"answer":null,"keyIdea":"Perplexity averages away exactly the tokens quantisation damages: measure with KL divergence against the original and with a task set containing exact-syntax, long-instruction and non-English items at temperature 0.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"Items that will show it","items":["JSON scored by a parser, not by eye","A prompt with twelve numbered constraints","A fact placed 8,000 tokens into the input","Items in every language you actually use","Arithmetic and rare API names"]},"type":"compare","right":{"title":"Items that will report nothing","items":["Open-ended English chat, graded by feel","Short questions with short answers","A perplexity delta from 6.21 to 6.24","Anything run once at temperature 0.7","Thirty items all from the easy case"]},"title":"A set that can detect the damage, and one that cannot","caption":"Perplexity averages over the vast majority of tokens every version of the model gets right, and the tokens quantisation damages are a tiny fraction of any corpus. Run the left-hand set at temperature 0 against the full-precision weights as well, or a quantisation failure and a model that simply cannot do the task look identical."},"alt":"Figure: A set that can detect the damage, and one that cannot. Items that will show it: JSON scored by a parser, not by eye; A prompt with twelve numbered constraints; A fact placed 8,000 tokens into the input; Items in every language you actually use; Arithmetic and rare API names. Items that will report nothing: Open-ended English chat, graded by feel; Short questions with short answers; A perplexity delta from 6.21 to 6.24; Anything run once at temperature 0.7; Thirty items all from the easy case. Perplexity averages over the vast majority of tokens every version of the model gets right, and the tokens quantisation damages are a tiny fraction of any corpus. Run the left-hand set at temperature 0 against the full-precision weights as well, or a quantisation failure and a model that simply cannot do the task look identical."}},"blocks":[{"t":"h2","id":"the-number-everybody-quotes-cannot-detect-the-problem","runs":[{"text":"The number everybody quotes cannot detect the problem"}]},{"t":"p","runs":[{"text":"Perplexity is the exponentiated average negative log-likelihood of a test corpus: how surprised the model is by text it should find ordinary. Every quantisation comparison publishes it, usually as a table of deltas: fp16 at 6.21, Q4_K_M at 6.24."}]},{"t":"p","runs":[{"text":"The trouble is what that average hides. Perplexity is dominated by the vast majority of tokens that are easy and that every version of the model gets right. The tokens where quantisation actually hurts (the closing brace, the correct API name, the eleventh constraint in a specification, the rare word in Tamil) are a tiny fraction of the corpus and move the average by almost nothing."}]},{"t":"p","runs":[{"text":"So a 0.5% perplexity delta is compatible with a model that has become clearly worse at code and clearly worse in one of its languages. Perplexity is not useless; it will catch a badly broken quant. It cannot rank two reasonable ones, and it is quoted as though it can."}]},{"t":"h2","id":"a-better-cheap-signal-divergence-from-the-original","runs":[{"text":"A better cheap signal: divergence from the original"}]},{"t":"p","runs":[{"text":"Instead of asking how surprised the quantised model is by a corpus, ask how differently it distributes probability compared with the model it came from. Feed the same tokens to both, and compare the full output distributions with KL divergence."}]},{"t":"code","lang":"bash","label":"Bash","text":"# generate a reference distribution file with the full-precision model\nllama-perplexity -m model-f16.gguf -f test.txt --kl-divergence-base ref.dat\n\n# then score the quant against it\nllama-perplexity -m model-Q4_K_M.gguf -f test.txt --kl-divergence ref.dat"},{"t":"p","runs":[{"text":"The useful outputs are the mean KL divergence and, more importantly, the top-token agreement rate and the 99th-percentile divergence. Mean divergence tells you the typical difference; the tail tells you how often the quant disagrees badly, which is the thing that produces a visibly wrong answer. Two quants with the same mean and different tails behave very differently in practice."}]},{"t":"p","runs":[{"text":"This is still a proxy. It measures difference from the original, not correctness, and a quant that diverges in a harmless direction scores the same as one that diverges in a harmful one."}]},{"t":"h2","id":"the-measurement-that-answers-your-actual-question","runs":[{"text":"The measurement that answers your actual question"}]},{"t":"p","runs":[{"text":"Your thirty-item audition set, run at temperature 0 against both the full-precision model and each candidate quant, graded blind."}]},{"t":"p","runs":[{"text":"Temperature 0 matters here: at any positive temperature you are measuring sampling noise as well as quantisation, and you will need many more items to see through it. Greedy decoding removes that variable entirely."}]},{"t":"p","runs":[{"text":"Design the set so it can detect what quantisation actually damages. Include:"}]},{"t":"ul","items":[[{"text":"Exact-syntax items.","b":true},{"text":" Valid JSON against a schema, a regex, a shell command, a piece of code that must compile. Score these automatically (parse the JSON, run the code) because they are objective and cheap."}],[{"text":"Long-instruction items.","b":true},{"text":" A prompt with twelve numbered constraints. Score by counting how many were honoured. This is the most sensitive test of quantisation damage there is and almost nobody runs it."}],[{"text":"Long-context items.","b":true},{"text":" A fact placed 8,000 tokens in, then asked about. Sensitive to KV-cache quantisation specifically."}],[{"text":"Items in every language you use.","b":true},{"text":" Quantisation damages low-resource languages first, and an English-only test will report that nothing happened."}],[{"text":"Arithmetic and rare-name items","b":true},{"text":", where a single wrong token is unambiguously wrong."}]]},{"t":"h2","id":"reading-a-result-honestly","runs":[{"text":"Reading a result honestly"}]},{"t":"p","runs":[{"text":"Thirty items is a small sample. A difference of one or two items between two quants is noise; a difference of eight is real. If the two candidates come out within a couple of items of each other, take the smaller or faster one and stop measuring: you have established that the difference is below your ability to detect, which is a legitimate and useful finding."}]},{"t":"p","runs":[{"text":"Run the same set against the "},{"text":"unquantised","i":true},{"text":" model too, even if you will never deploy it. Without that reference you cannot tell a quantisation failure from a model that cannot do the task, and those two conclusions lead to completely different next steps."}]},{"t":"h2","id":"what-to-write-down","runs":[{"text":"What to write down"}]},{"t":"p","runs":[{"text":"Model, quant, engine version, context length, sampler settings, the date, the scores. Engines change, defaults change, and a quant that was fine in one version can regress in the next. Six lines in a text file, and the next time somebody claims a quant is bad you can answer with a measurement rather than an impression."}]}],"check":{"question":[{"text":"A quant shows a perplexity increase of 0.4% and a mean KL divergence against fp16 that looks small, yet users report it producing invalid JSON far more often. Which measurement would most likely have predicted this?"}],"options":[{"text":[{"text":"Perplexity measured on a corpus of JSON documents rather than on general prose."}],"correct":false,"tempting":[{"text":"Moves the corpus but keeps the averaging that hides rare critical tokens; most JSON tokens are as easy as most prose tokens."}]},{"text":[{"text":"A larger evaluation corpus, since 0.4% is within noise at typical perplexity test sizes."}],"correct":false,"tempting":[{"text":"Treats a systematic blind spot as a sample-size issue; more of the same measurement reports the same average more precisely."}]},{"text":[{"text":"The tail of the divergence distribution and a schema-validated generation test, since the failure is rare disagreements on structurally critical tokens rather than a shift in the typical case."}],"correct":true,"tempting":null},{"text":[{"text":"Top-token agreement rate on the same corpus, which would have shown the model choosing different tokens overall."}],"correct":false,"tempting":[{"text":"Closer, but still an aggregate over ordinary text; agreement can stay high while the few decisive tokens flip."}]}],"order":[0,1,3,2],"why":null},"prev":{"slug":"quantising-a-model-yourself","title":"Quantising a model yourself","minutes":10},"next":{"slug":"the-size-precision-frontier","title":"Bigger and crushed, or smaller and clean","minutes":8}}}