{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"comparing-quantisations","title":"Proving what a quantisation cost","url":"https://addaly.com/learn/open-models/comparing-quantisations","position":66,"of":71,"minutes":9,"answer":"Perplexity is a screening test; the sharper measurement is divergence from the full-precision model on the same corpus, reported as the fraction of tokens where the top choice differs.","keyIdea":"Perplexity is a screening test; the sharper measurement is divergence from the full-precision model on the same corpus, reported as the fraction of tokens where the top choice differs, plus agreement on your own examples at long context and in your languages."},"body":{"format":"markdown","text":"## Four methods, from weakest to strongest\n\nYou have a Q4 file and a claim that it is nearly lossless. Here is how to check, in ascending order of usefulness.\n\n## 1. Perplexity\n\n```bash\n./llama-perplexity -m model-f16.gguf -f wiki.test.raw -c 4096\n./llama-perplexity -m model-Q4_K_M.gguf -f wiki.test.raw -c 4096\n```\n\nPerplexity is the exponentiated average negative log-likelihood on a fixed corpus: how surprised the model is by text it did not write. Lower is better, and the comparison is only meaningful between two models sharing a tokenizer, which is exactly the quantisation case.\n\nA rise from 6.10 to 6.18 is small. A rise to 7.4 is not. But two quantisations can sit within a hundredth of each other and behave differently on structured output, so perplexity is a screening test, not a verdict.\n\n## 2. Divergence from the reference\n\nMuch more sensitive, and the right default. Instead of asking how surprised each model is by a corpus, ask **how far the quantised model's output distribution has moved from the full-precision model's**.\n\n```bash\n# produce a reference from the unquantised model\n./llama-perplexity -m model-f16.gguf -f wiki.test.raw --kl-divergence-base ref.dat\n\n# compare a quantisation against it\n./llama-perplexity -m model-Q4_K_M.gguf -f wiki.test.raw --kl-divergence ref.dat\n```\n\nThe output includes the mean Kullback-Leibler divergence and, more usefully, the **fraction of tokens where the top choice differs** between the two models. That last figure is directly interpretable: if the quantised model picks a different most-likely token 1.5% of the time, you know what you bought. Above about 5%, expect visible behavioural differences.\n\nReport the tail as well as the mean. A quantisation with a low mean divergence and a heavy 99th percentile is one that is usually identical and occasionally very different, which is worse for a product than a uniform small degradation.\n\n## 3. Your own task\n\nRun your forty held-out examples through both files at temperature 0, and compare the answers directly. Two numbers matter: the task score for each, and the **agreement rate** between them.\n\nAgreement is the sharper instrument on small sets. With forty examples, a two-point score difference is noise; but if the two models produce different answers on nine of forty items, something real has changed even when the scores happen to match.\n\n## 4. The specific capabilities quantisation damages\n\nFrom the earlier lesson: long-context retrieval, lower-resource languages, multi-step arithmetic and long-output instruction adherence degrade before short English chat does. So test those deliberately rather than hoping a general set covers them.\n\n- Ten questions whose answers sit deep in a long document, at your real context length.\n- Ten items in each non-English language you serve.\n- Ten items requiring a strict output format held over a long response.\n\nThis is thirty items and it is where the difference will appear if there is one.\n\n## Test the cache separately\n\nKV cache quantisation is a different decision from weight quantisation and is frequently changed at the same time, which confounds both. Run four configurations if the setting matters to you: full weights with full cache, quantised weights with full cache, full weights with quantised cache, and both. The long-context items are where cache quantisation shows up, and it is often the cause when a model that tested well behaves oddly on long documents.\n\n## What to write down\n\nFor any quantisation you ship:\n\n```\nmodel      Qwen3-8B @ 9a3f21c\nquant      Q4_K_M (imatrix, bartowski, rev 2f8c1)\nperplexity 6.10 -> 6.17 on wiki.test.raw, c=4096\ntop-token  differs on 1.4% of tokens vs f16\ntask       0.86 -> 0.85 on held-out v4 (40 items), agreement 37/40\nlong-ctx   0.90 -> 0.80 on the 10 deep-retrieval items\ndecision   accepted; long-context path routed to Q6\n```\n\nSix lines, and it turns a choice somebody made once into a documented decision anybody can revisit."},"check":{"question":"Two quantisations of the same model have almost identical perplexity, but one changes the model's top-choice token on 0.9% of positions and the other on 6%. What follows?","options":[{"text":"Perplexity is the more reliable metric, so the two quantisations are equivalent in practice.","correct":false,"tempting":"Perplexity averages log-likelihood over all tokens and can hide large disagreement about which token wins."},{"text":"The 6% quantisation will behave visibly differently in generation, and perplexity failed to separate them because it averages likelihood rather than tracking the chosen token.","correct":true},{"text":"The 6% figure indicates a corrupted file, since a valid quantisation cannot differ that often.","correct":false,"tempting":"Aggressive but valid quantisations routinely reach that range; it is a quality cost, not corruption."},{"text":"Top-token disagreement is expected to be high at any bit width because sampling is stochastic.","correct":false,"tempting":"The measurement compares distributions rather than samples, so stochastic sampling does not enter into it."}]},"prev":{"slug":"running-the-eval-harness","title":"Running the standard harness yourself","url":"https://addaly.com/learn/open-models/running-the-eval-harness"},"next":{"slug":"evaluating-an-embedding-model","title":"Evaluating an embedding model on your corpus","url":"https://addaly.com/learn/open-models/evaluating-an-embedding-model"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Proving what a quantisation cost', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/comparing-quantisations (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Proving what a quantisation cost","courseTitle":"The Open Model Ecosystem","module":{"index":7,"slug":"judging-models-and-claims","title":"Judging models and claims"},"answer":null,"keyIdea":"Perplexity is a screening test; the sharper measurement is divergence from the full-precision model on the same corpus, reported as the fraction of tokens where the top choice differs, plus agreement on your own examples at long context and in your languages.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"four-methods-from-weakest-to-strongest","runs":[{"text":"Four methods, from weakest to strongest"}]},{"t":"p","runs":[{"text":"You have a Q4 file and a claim that it is nearly lossless. Here is how to check, in ascending order of usefulness."}]},{"t":"h2","id":"1-perplexity","runs":[{"text":"1. Perplexity"}]},{"t":"code","lang":"bash","label":"Bash","text":"./llama-perplexity -m model-f16.gguf -f wiki.test.raw -c 4096\n./llama-perplexity -m model-Q4_K_M.gguf -f wiki.test.raw -c 4096"},{"t":"p","runs":[{"text":"Perplexity is the exponentiated average negative log-likelihood on a fixed corpus: how surprised the model is by text it did not write. Lower is better, and the comparison is only meaningful between two models sharing a tokenizer, which is exactly the quantisation case."}]},{"t":"p","runs":[{"text":"A rise from 6.10 to 6.18 is small. A rise to 7.4 is not. But two quantisations can sit within a hundredth of each other and behave differently on structured output, so perplexity is a screening test, not a verdict."}]},{"t":"h2","id":"2-divergence-from-the-reference","runs":[{"text":"2. Divergence from the reference"}]},{"t":"p","runs":[{"text":"Much more sensitive, and the right default. Instead of asking how surprised each model is by a corpus, ask "},{"text":"how far the quantised model’s output distribution has moved from the full-precision model’s","b":true},{"text":"."}]},{"t":"code","lang":"bash","label":"Bash","text":"# produce a reference from the unquantised model\n./llama-perplexity -m model-f16.gguf -f wiki.test.raw --kl-divergence-base ref.dat\n\n# compare a quantisation against it\n./llama-perplexity -m model-Q4_K_M.gguf -f wiki.test.raw --kl-divergence ref.dat"},{"t":"p","runs":[{"text":"The output includes the mean Kullback-Leibler divergence and, more usefully, the "},{"text":"fraction of tokens where the top choice differs","b":true},{"text":" between the two models. That last figure is directly interpretable: if the quantised model picks a different most-likely token 1.5% of the time, you know what you bought. Above about 5%, expect visible behavioural differences."}]},{"t":"p","runs":[{"text":"Report the tail as well as the mean. A quantisation with a low mean divergence and a heavy 99th percentile is one that is usually identical and occasionally very different, which is worse for a product than a uniform small degradation."}]},{"t":"h2","id":"3-your-own-task","runs":[{"text":"3. Your own task"}]},{"t":"p","runs":[{"text":"Run your forty held-out examples through both files at temperature 0, and compare the answers directly. Two numbers matter: the task score for each, and the "},{"text":"agreement rate","b":true},{"text":" between them."}]},{"t":"p","runs":[{"text":"Agreement is the sharper instrument on small sets. With forty examples, a two-point score difference is noise; but if the two models produce different answers on nine of forty items, something real has changed even when the scores happen to match."}]},{"t":"h2","id":"4-the-specific-capabilities-quantisation-damages","runs":[{"text":"4. The specific capabilities quantisation damages"}]},{"t":"p","runs":[{"text":"From the earlier lesson: long-context retrieval, lower-resource languages, multi-step arithmetic and long-output instruction adherence degrade before short English chat does. So test those deliberately rather than hoping a general set covers them."}]},{"t":"ul","items":[[{"text":"Ten questions whose answers sit deep in a long document, at your real context length."}],[{"text":"Ten items in each non-English language you serve."}],[{"text":"Ten items requiring a strict output format held over a long response."}]]},{"t":"p","runs":[{"text":"This is thirty items and it is where the difference will appear if there is one."}]},{"t":"h2","id":"test-the-cache-separately","runs":[{"text":"Test the cache separately"}]},{"t":"p","runs":[{"text":"KV cache quantisation is a different decision from weight quantisation and is frequently changed at the same time, which confounds both. Run four configurations if the setting matters to you: full weights with full cache, quantised weights with full cache, full weights with quantised cache, and both. The long-context items are where cache quantisation shows up, and it is often the cause when a model that tested well behaves oddly on long documents."}]},{"t":"h2","id":"what-to-write-down","runs":[{"text":"What to write down"}]},{"t":"p","runs":[{"text":"For any quantisation you ship:"}]},{"t":"code","lang":"","label":"Text","text":"model      Qwen3-8B @ 9a3f21c\nquant      Q4_K_M (imatrix, bartowski, rev 2f8c1)\nperplexity 6.10 -> 6.17 on wiki.test.raw, c=4096\ntop-token  differs on 1.4% of tokens vs f16\ntask       0.86 -> 0.85 on held-out v4 (40 items), agreement 37/40\nlong-ctx   0.90 -> 0.80 on the 10 deep-retrieval items\ndecision   accepted; long-context path routed to Q6"},{"t":"p","runs":[{"text":"Six lines, and it turns a choice somebody made once into a documented decision anybody can revisit."}]}],"check":{"question":[{"text":"Two quantisations of the same model have almost identical perplexity, but one changes the model’s top-choice token on 0.9% of positions and the other on 6%. What follows?"}],"options":[{"text":[{"text":"Perplexity is the more reliable metric, so the two quantisations are equivalent in practice."}],"correct":false,"tempting":[{"text":"Perplexity averages log-likelihood over all tokens and can hide large disagreement about which token wins."}]},{"text":[{"text":"The 6% quantisation will behave visibly differently in generation, and perplexity failed to separate them because it averages likelihood rather than tracking the chosen token."}],"correct":true,"tempting":null},{"text":[{"text":"The 6% figure indicates a corrupted file, since a valid quantisation cannot differ that often."}],"correct":false,"tempting":[{"text":"Aggressive but valid quantisations routinely reach that range; it is a quality cost, not corruption."}]},{"text":[{"text":"Top-token disagreement is expected to be high at any bit width because sampling is stochastic."}],"correct":false,"tempting":[{"text":"The measurement compares distributions rather than samples, so stochastic sampling does not enter into it."}]}],"order":[2,1,3,0],"why":null},"prev":{"slug":"running-the-eval-harness","title":"Running the standard harness yourself","minutes":9},"next":{"slug":"evaluating-an-embedding-model","title":"Evaluating an embedding model on your corpus","minutes":9}}}