{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"quantisation-mechanics","title":"Quantisation: what the numbers lose","url":"https://addaly.com/learn/how-llms-work/quantisation-mechanics","position":61,"of":78,"minutes":10,"answer":"Quantisation works because rounding errors average out across long dot products, which is exactly why it fails where they cannot (small models, multi-step arithmetic.","keyIdea":"Quantisation works because rounding errors average out across long dot products, which is exactly why it fails where they cannot (small models, multi-step arithmetic, long-context recall and weakly represented languages) and why perplexity is the wrong number to choose a level by."},"body":{"format":"markdown","text":"## What a quantised number actually is\n\nA bf16 weight has sixteen bits: about three significant decimal digits and an enormous range. An int8 has 256 possible values. A four-bit integer has sixteen. Quantisation is the act of mapping each weight onto one of those few values in a way that loses as little as possible.\n\nThe simplest scheme stores one **scale** per group of weights. Take four weights: 0.31, −0.12, 0.05, 0.90. For signed four-bit integers the range is −8 to 7, so the scale is the largest magnitude divided by seven: 0.90 ÷ 7 ≈ 0.129. Each weight becomes `round(w ÷ scale)`: 2, −1, 0, 7. To use them, multiply back: 0.26, −0.13, 0.00, 0.90.\n\nThe first weight has moved from 0.31 to 0.26, a 15 per cent error. That sounds unusable. It is not, because a weight is never used alone: it participates in a dot product over thousands of elements, and rounding errors in different directions largely cancel. The mechanism that makes quantisation work at all is averaging, which also predicts when it will fail: small models, with fewer weights per dot product, degrade more than large ones at the same bit width. They do.\n\n## Groups, and the outlier problem\n\nIf one scale covered a whole matrix, a single large value would ruin it. Suppose most weights are around 0.3 and one is 12. The scale becomes 12 ÷ 7 ≈ 1.7, and every ordinary weight rounds to zero. So scales are stored per group of 32 to 128 weights, in 16-bit. That is where the \"4.5 bits per weight\" figures come from: four bits of value and a share of a shared scale.\n\nWeights are the easy half. In 2022 it was found that models above roughly six billion parameters develop a few hidden dimensions whose **activations** are twenty to a hundred times larger than the rest. Naive 8-bit activations collapsed on such models. The fix was to keep those few columns in 16-bit and quantise everything else: a mixed decomposition. This is why the common local recipe is four-bit weights with 16-bit activations, written W4A16, and why servers that quantise activations to 8-bit prefer the FP8 format, which has an exponent and copes with outliers where a plain integer cannot.\n\n## Better than rounding\n\nRounding each weight to its nearest level ignores which weights matter. Two methods do better, both free, both standard.\n\n**GPTQ** quantises a matrix column by column, running a hundred or so calibration examples through the model, and after each column adjusts the not-yet-quantised weights to compensate for the error just introduced. The compensation uses second-order information about the loss, and the result is a four-bit model markedly closer to the original than rounding gives.\n\n**AWQ** observes that a small fraction of weight channels (around one per cent) carry most of the signal, identifiable from activation magnitudes on calibration data. It scales those channels up before quantising so they receive more of the available resolution, and folds the scaling into the next layer.\n\nIn the GGUF world that `llama.cpp` and Ollama use, the names encode the recipe. `Q8_0` is 8-bit and effectively lossless. `Q4_K_M` is four-bit with grouped scales in a superblock layout, with a few sensitive tensors (attention values, the output projection) kept at higher precision; it is the default trade almost everyone lands on. `Q2` and `Q3` variants visibly degrade. Files marked `IQ` or built with an \"importance matrix\" used calibration data in the AWQ spirit.\n\n## What degrades, and where it hides\n\nMeasured on perplexity (module five's loss, exponentiated) an 8-billion model at `Q4_K_M` is typically one to three per cent worse than bf16, and at 8-bit it is indistinguishable. That number is quoted everywhere and it hides the damage.\n\nPerplexity averages over every token of ordinary text, most of which is easy. The loss from quantisation concentrates where the margins were already thin:\n\n> Figure: Where the four-bit loss actually lands. Bars (points lost against bf16, same eight-billion model): Perplexity on ordinary text, the quoted number 2; General knowledge questions 3; Precise instruction-following and formatting 6; Long-context recall, with a 4-bit cache 9; Multi-step arithmetic and code 10. The averaged number hides the damage, because most tokens of ordinary text are easy. The loss concentrates where the margins were already thin: steps that compound, recall over long inputs, precise formats, and the low-resource languages of module six, which had the least margin to begin with.\n\n- **Multi-step arithmetic and code**, where small per-step errors compound across steps. A model two per cent worse on perplexity can be ten points worse on a maths benchmark at four bits.\n- **Long-context recall**, especially if the cache is quantised too. 8-bit cache is close to free; 4-bit cache costs measurable needle-finding accuracy.\n- **Low-resource languages.** Module six showed those directions were weakly represented to begin with; they have the least margin to lose.\n- **Precise instruction-following** (formatting constraints, exact lengths) which lives in the same thin margins.\n\nLarger models tolerate low bit widths better, for the averaging reason. A four-bit 70-billion model is usually excellent. A four-bit 1-billion model is often noticeably damaged, and 3-bit versions of small models are frequently not worth running.\n\n## The quantisation you cannot see\n\nProviders quantise too. FP8 serving on current data-centre hardware is routine, and serving-side precision is one of the things that can change under a fixed model name. It is one of the signatures in module six's drift lesson: a small, broad degradation, worst on arithmetic and long-context tasks, appearing on a date.\n\n## The test that replaces the perplexity number\n\nDo not choose a quantisation level from a chart. Take your own hundred-item evaluation set, run it at bf16 and at the candidate quantisation, and compare: per task, not overall. If the drop sits inside the noise band you measured in module six, take the speed and the memory. If it does not, try the next level up, or a bigger model at the lower precision.\n\nThat last comparison matters more than it looks. A four-bit 14-billion model usually beats a bf16 7-billion model on quality and costs about the same memory. A 2-bit 70-billion model often loses to a four-bit 32-billion model. Quantisation is one axis of a two-axis trade with model size, and the right point is found by measuring, in an afternoon, with free tools."},"check":{"question":"A 7-billion-parameter model quantised to four bits shows perplexity only 2 per cent worse than bf16, yet on a team's multi-step arithmetic set accuracy falls from 71 to 58 per cent. What explains the mismatch?","options":[{"text":"Perplexity was measured on a general text corpus rather than on the team's own data, so it is simply the wrong measurement to compare against.","correct":false,"tempting":"Half right (perplexity is the wrong number), but for the reason of what it averages over, not where it was measured; perplexity on the team's own text would hide the same damage."},{"text":"Thirteen points on a small evaluation set is inside ordinary sampling noise, so nothing has actually been shown to degrade.","correct":false,"tempting":"Dismisses a systematic, well-documented direction of loss as noise; measured against the noise band the drop is large and reproducible."},{"text":"Digit tokens are stored at lower precision than word tokens in a four-bit file, so anything involving numbers loses more than text does.","correct":false,"tempting":"Invents a distinction the format does not make; weights are quantised by matrix and group, with no knowledge of which tokens they will later serve."},{"text":"Perplexity averages over every easy token of ordinary text, while multi-step tasks compound small per-step errors.","correct":true}]},"prev":{"slug":"will-it-fit","title":"Will it fit: the memory budget of a running model","url":"https://addaly.com/learn/how-llms-work/will-it-fit"},"next":{"slug":"running-on-your-own-machine","title":"Running a model on your own machine","url":"https://addaly.com/learn/how-llms-work/running-on-your-own-machine"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Quantisation: what the numbers lose', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/quantisation-mechanics (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Quantisation: what the numbers lose","courseTitle":"How a Language Model Actually Works","module":{"index":7,"slug":"running-it","title":"Running it: memory, speed and money"},"answer":null,"keyIdea":"Quantisation works because rounding errors average out across long dot products, which is exactly why it fails where they cannot (small models, multi-step arithmetic, long-context recall and weakly represented languages) and why perplexity is the wrong number to choose a level by.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"points lost against bf16, same eight-billion model","title":"Where the four-bit loss actually lands","series":[{"label":"Perplexity on ordinary text, the quoted number","value":2},{"label":"General knowledge questions","value":3},{"label":"Precise instruction-following and formatting","value":6},{"label":"Long-context recall, with a 4-bit cache","value":9},{"label":"Multi-step arithmetic and code","value":10,"highlight":true}],"caption":"The averaged number hides the damage, because most tokens of ordinary text are easy. The loss concentrates where the margins were already thin: steps that compound, recall over long inputs, precise formats, and the low-resource languages of module six, which had the least margin to begin with."},"alt":"Figure: Where the four-bit loss actually lands. Bars (points lost against bf16, same eight-billion model): Perplexity on ordinary text, the quoted number 2; General knowledge questions 3; Precise instruction-following and formatting 6; Long-context recall, with a 4-bit cache 9; Multi-step arithmetic and code 10. The averaged number hides the damage, because most tokens of ordinary text are easy. The loss concentrates where the margins were already thin: steps that compound, recall over long inputs, precise formats, and the low-resource languages of module six, which had the least margin to begin with."}},"blocks":[{"t":"h2","id":"what-a-quantised-number-actually-is","runs":[{"text":"What a quantised number actually is"}]},{"t":"p","runs":[{"text":"A bf16 weight has sixteen bits: about three significant decimal digits and an enormous range. An int8 has 256 possible values. A four-bit integer has sixteen. Quantisation is the act of mapping each weight onto one of those few values in a way that loses as little as possible."}]},{"t":"p","runs":[{"text":"The simplest scheme stores one "},{"text":"scale","b":true},{"text":" per group of weights. Take four weights: 0.31, −0.12, 0.05, 0.90. For signed four-bit integers the range is −8 to 7, so the scale is the largest magnitude divided by seven: 0.90 ÷ 7 ≈ 0.129. Each weight becomes "},{"text":"round(w ÷ scale)","code":true},{"text":": 2, −1, 0, 7. To use them, multiply back: 0.26, −0.13, 0.00, 0.90."}]},{"t":"p","runs":[{"text":"The first weight has moved from 0.31 to 0.26, a 15 per cent error. That sounds unusable. It is not, because a weight is never used alone: it participates in a dot product over thousands of elements, and rounding errors in different directions largely cancel. The mechanism that makes quantisation work at all is averaging, which also predicts when it will fail: small models, with fewer weights per dot product, degrade more than large ones at the same bit width. They do."}]},{"t":"h2","id":"groups-and-the-outlier-problem","runs":[{"text":"Groups, and the outlier problem"}]},{"t":"p","runs":[{"text":"If one scale covered a whole matrix, a single large value would ruin it. Suppose most weights are around 0.3 and one is 12. The scale becomes 12 ÷ 7 ≈ 1.7, and every ordinary weight rounds to zero. So scales are stored per group of 32 to 128 weights, in 16-bit. That is where the “4.5 bits per weight” figures come from: four bits of value and a share of a shared scale."}]},{"t":"p","runs":[{"text":"Weights are the easy half. In 2022 it was found that models above roughly six billion parameters develop a few hidden dimensions whose "},{"text":"activations","b":true},{"text":" are twenty to a hundred times larger than the rest. Naive 8-bit activations collapsed on such models. The fix was to keep those few columns in 16-bit and quantise everything else: a mixed decomposition. This is why the common local recipe is four-bit weights with 16-bit activations, written W4A16, and why servers that quantise activations to 8-bit prefer the FP8 format, which has an exponent and copes with outliers where a plain integer cannot."}]},{"t":"h2","id":"better-than-rounding","runs":[{"text":"Better than rounding"}]},{"t":"p","runs":[{"text":"Rounding each weight to its nearest level ignores which weights matter. Two methods do better, both free, both standard."}]},{"t":"p","runs":[{"text":"GPTQ","b":true},{"text":" quantises a matrix column by column, running a hundred or so calibration examples through the model, and after each column adjusts the not-yet-quantised weights to compensate for the error just introduced. The compensation uses second-order information about the loss, and the result is a four-bit model markedly closer to the original than rounding gives."}]},{"t":"p","runs":[{"text":"AWQ","b":true},{"text":" observes that a small fraction of weight channels (around one per cent) carry most of the signal, identifiable from activation magnitudes on calibration data. It scales those channels up before quantising so they receive more of the available resolution, and folds the scaling into the next layer."}]},{"t":"p","runs":[{"text":"In the GGUF world that "},{"text":"llama.cpp","code":true},{"text":" and Ollama use, the names encode the recipe. "},{"text":"Q8_0","code":true},{"text":" is 8-bit and effectively lossless. "},{"text":"Q4_K_M","code":true},{"text":" is four-bit with grouped scales in a superblock layout, with a few sensitive tensors (attention values, the output projection) kept at higher precision; it is the default trade almost everyone lands on. "},{"text":"Q2","code":true},{"text":" and "},{"text":"Q3","code":true},{"text":" variants visibly degrade. Files marked "},{"text":"IQ","code":true},{"text":" or built with an “importance matrix” used calibration data in the AWQ spirit."}]},{"t":"h2","id":"what-degrades-and-where-it-hides","runs":[{"text":"What degrades, and where it hides"}]},{"t":"p","runs":[{"text":"Measured on perplexity (module five’s loss, exponentiated) an 8-billion model at "},{"text":"Q4_K_M","code":true},{"text":" is typically one to three per cent worse than bf16, and at 8-bit it is indistinguishable. That number is quoted everywhere and it hides the damage."}]},{"t":"p","runs":[{"text":"Perplexity averages over every token of ordinary text, most of which is easy. The loss from quantisation concentrates where the margins were already thin:"}]},{"t":"ul","items":[[{"text":"Multi-step arithmetic and code","b":true},{"text":", where small per-step errors compound across steps. A model two per cent worse on perplexity can be ten points worse on a maths benchmark at four bits."}],[{"text":"Long-context recall","b":true},{"text":", especially if the cache is quantised too. 8-bit cache is close to free; 4-bit cache costs measurable needle-finding accuracy."}],[{"text":"Low-resource languages.","b":true},{"text":" Module six showed those directions were weakly represented to begin with; they have the least margin to lose."}],[{"text":"Precise instruction-following","b":true},{"text":" (formatting constraints, exact lengths) which lives in the same thin margins."}]]},{"t":"p","runs":[{"text":"Larger models tolerate low bit widths better, for the averaging reason. A four-bit 70-billion model is usually excellent. A four-bit 1-billion model is often noticeably damaged, and 3-bit versions of small models are frequently not worth running."}]},{"t":"h2","id":"the-quantisation-you-cannot-see","runs":[{"text":"The quantisation you cannot see"}]},{"t":"p","runs":[{"text":"Providers quantise too. FP8 serving on current data-centre hardware is routine, and serving-side precision is one of the things that can change under a fixed model name. It is one of the signatures in module six’s drift lesson: a small, broad degradation, worst on arithmetic and long-context tasks, appearing on a date."}]},{"t":"h2","id":"the-test-that-replaces-the-perplexity-number","runs":[{"text":"The test that replaces the perplexity number"}]},{"t":"p","runs":[{"text":"Do not choose a quantisation level from a chart. Take your own hundred-item evaluation set, run it at bf16 and at the candidate quantisation, and compare: per task, not overall. If the drop sits inside the noise band you measured in module six, take the speed and the memory. If it does not, try the next level up, or a bigger model at the lower precision."}]},{"t":"p","runs":[{"text":"That last comparison matters more than it looks. A four-bit 14-billion model usually beats a bf16 7-billion model on quality and costs about the same memory. A 2-bit 70-billion model often loses to a four-bit 32-billion model. Quantisation is one axis of a two-axis trade with model size, and the right point is found by measuring, in an afternoon, with free tools."}]}],"check":{"question":[{"text":"A 7-billion-parameter model quantised to four bits shows perplexity only 2 per cent worse than bf16, yet on a team’s multi-step arithmetic set accuracy falls from 71 to 58 per cent. What explains the mismatch?"}],"options":[{"text":[{"text":"Perplexity was measured on a general text corpus rather than on the team’s own data, so it is simply the wrong measurement to compare against."}],"correct":false,"tempting":[{"text":"Half right (perplexity is the wrong number), but for the reason of what it averages over, not where it was measured; perplexity on the team’s own text would hide the same damage."}]},{"text":[{"text":"Thirteen points on a small evaluation set is inside ordinary sampling noise, so nothing has actually been shown to degrade."}],"correct":false,"tempting":[{"text":"Dismisses a systematic, well-documented direction of loss as noise; measured against the noise band the drop is large and reproducible."}]},{"text":[{"text":"Digit tokens are stored at lower precision than word tokens in a four-bit file, so anything involving numbers loses more than text does."}],"correct":false,"tempting":[{"text":"Invents a distinction the format does not make; weights are quantised by matrix and group, with no knowledge of which tokens they will later serve."}]},{"text":[{"text":"Perplexity averages over every easy token of ordinary text, while multi-step tasks compound small per-step errors."}],"correct":true,"tempting":null}],"order":[2,1,3,0],"why":null},"prev":{"slug":"will-it-fit","title":"Will it fit: the memory budget of a running model","minutes":9},"next":{"slug":"running-on-your-own-machine","title":"Running a model on your own machine","minutes":9}}}