{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"quantisation-explained","title":"What quantisation actually does","url":"https://addaly.com/learn/open-models/quantisation-explained","position":32,"of":71,"minutes":10,"answer":"Quantisation stores weights as small integers with a shared scale per block, so a 4-bit model is really about 5 bits per weight, and because larger models carry more redundancy.","keyIdea":"Quantisation stores weights as small integers with a shared scale per block, so a 4-bit model is really about 5 bits per weight, and because larger models carry more redundancy, a bigger model quantised harder usually beats a smaller one at full precision for the same memory."},"body":{"format":"markdown","text":"## Storing a number in four bits\n\nA weight in a released model is usually bfloat16: sixteen bits, of which eight are exponent. Quantisation stores it in fewer bits, and the whole art is in how you keep the information that matters.\n\nNaively rounding every weight to one of sixteen values across the model's full range would be a disaster, because a handful of large weights would stretch the range and everything else would collapse to zero. So quantisation is done **block-wise**. Split the weights into blocks of 32 or 64. For each block, find the largest magnitude, store a scale factor, and store each weight as a small integer index into that block's range.\n\n```\nblock of 32 weights, 4-bit:\n  32 indices x 4 bits           = 128 bits\n  1 scale in fp16               =  16 bits\n  (some schemes also a minimum) =  16 bits\n  ------------------------------------------\n  160 bits / 32 weights          = 5.0 bits per weight\n```\n\nThis is why a model described as 4-bit is not exactly half the size of an 8-bit one. Real formats land between 4.5 and 5.5 bits per weight once the scales are counted, which is exactly why an 8B model at Q4_K_M is about 4.9 GB rather than 4.0 GB. If you have ever wondered where that extra gigabyte came from, it is the scales.\n\n> Figure: An 8B model on disk, by quantisation. Bars (GB): bf16, the reference 16.1; Q8_0 8.5; Q6_K 6.6; Q5_K_M 5.7; Q4_K_M 4.9; Q3_K_M 3.9; Q2_K 3. Q4_K_M is about 4.9 GB rather than 4.0 because each block of weights carries a scale, and often a minimum, alongside the indices: roughly five bits per weight, not four. That extra gigabyte is the scales, and it is the answer to where it came from.\n\n## The K in K-quants\n\nllama.cpp's K-quant family does something smarter than uniform treatment: it assigns different precision to different tensors. The layers that hurt most when degraded (in practice, parts of the attention and one of the feed-forward matrices) keep more bits, while the rest take fewer. The suffixes encode this. `Q4_K_S` is the small variant, `Q4_K_M` the medium, and the M version spends its extra half-bit where measurements said it mattered.\n\nMore recent work adds an **importance matrix**: run a calibration corpus through the model, record which weights have the largest effect on the output, and let the quantiser protect them. Files named with `imatrix` were built this way, and the benefit is largest at very low bit widths, where there is least room for error.\n\n## The other methods, and what each assumes\n\n**Round to nearest** is the baseline: no calibration, instant, and the weakest.\n\n**GPTQ** processes weights layer by layer, using a small calibration dataset and second-order information to compensate the remaining weights for each rounding error as it goes. Slow to produce, good results, GPU-oriented.\n\n**AWQ** starts from an observation: roughly 1% of weight channels matter far more than the rest, and which ones can be identified from the *activations* rather than the weights. Protect those channels by scaling, quantise the rest hard. Fast and strong, and widely used for GPU serving.\n\n**bitsandbytes NF4** uses a 4-bit format shaped for normally distributed values, which weights approximately are. It is the format under QLoRA fine-tuning, and it is designed to be loaded and trained against rather than to be maximally compact.\n\n**FP8** is not really quantisation in the same sense: it is a hardware-supported floating point format on recent Nvidia cards, giving near-lossless quality at half the memory with no calibration. Where the hardware supports it, it is the easy win.\n\n## Why activation outliers matter\n\nThe reason naive 8-bit integer quantisation failed on transformers, and the reason the methods above exist, is a specific empirical fact: in large models a few hidden dimensions carry activations with magnitudes hundreds of times larger than the rest. Quantise a matrix multiplication that includes those channels uniformly and the small values disappear entirely.\n\nEvery serious method handles this somehow: by keeping outlier channels in higher precision, by scaling them before quantising, or by moving the difficulty from activations into weights. It is worth knowing because it explains why quantisation quality is not a smooth function of bit width and why methods that ignore the phenomenon fail suddenly rather than gradually.\n\n## What it costs, in one sentence you can act on\n\nFor models above about 7 billion parameters, going from 16-bit to a good 4-bit scheme typically costs a small fraction of a point on most tasks: usually invisible against the noise of a 40-example evaluation set.\n\nWhich produces the rule that decides most local setups: **a larger model quantised harder usually beats a smaller model at full precision**, for the same memory. A 13B at 4 bits and an 7B at 8 bits both occupy about 8 GB, and the 13B is generally better.\n\nThe rule has limits, and they matter. Below about 3 billion parameters the model has less redundancy to spare and quantisation hurts more. Quantisation degrades long-context retrieval and non-English performance more than it degrades short English chat, which is precisely the sort of thing a general benchmark hides. And below 4 bits the curve steepens sharply. Those limits are the subject of the next lesson."},"check":{"question":"Why is an 8B model in Q4_K_M about 4.9 GB rather than the 4.0 GB that four bits per weight would predict?","options":[{"text":"The tokenizer and configuration files add roughly a gigabyte to the download.","correct":false,"tempting":"Those files are a few megabytes; they cannot account for a gigabyte."},{"text":"Quantisation stores per-block scale factors alongside the indices, and K-quants keep selected tensors at higher precision, which together push the effective rate to about five bits per weight.","correct":true},{"text":"The embedding matrices cannot be quantised and remain at 16 bits.","correct":false,"tempting":"Embeddings are often kept at higher precision than the average, but they are quantised too and are not the main cause."},{"text":"GGUF files include the KV cache allocation in the file size.","correct":false,"tempting":"The cache is allocated at runtime from your memory and is not stored in the model file at all."}]},"prev":{"slug":"choosing-for-the-job","title":"Choosing a Model for the Job","url":"https://addaly.com/learn/open-models/choosing-for-the-job"},"next":{"slug":"choosing-a-bit-width","title":"Choosing a bit width you can defend","url":"https://addaly.com/learn/open-models/choosing-a-bit-width"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What quantisation actually does', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/quantisation-explained (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What quantisation actually does","courseTitle":"The Open Model Ecosystem","module":{"index":4,"slug":"making-it-run","title":"Making it run on the machine you have"},"answer":null,"keyIdea":"Quantisation stores weights as small integers with a shared scale per block, so a 4-bit model is really about 5 bits per weight, and because larger models carry more redundancy, a bigger model quantised harder usually beats a smaller one at full precision for the same memory.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"GB","title":"An 8B model on disk, by quantisation","series":[{"label":"bf16, the reference","value":16.1},{"label":"Q8_0","value":8.5},{"label":"Q6_K","value":6.6},{"label":"Q5_K_M","value":5.7},{"label":"Q4_K_M","value":4.9,"highlight":true},{"label":"Q3_K_M","value":3.9},{"label":"Q2_K","value":3}],"caption":"Q4_K_M is about 4.9 GB rather than 4.0 because each block of weights carries a scale, and often a minimum, alongside the indices: roughly five bits per weight, not four. That extra gigabyte is the scales, and it is the answer to where it came from."},"alt":"Figure: An 8B model on disk, by quantisation. Bars (GB): bf16, the reference 16.1; Q8_0 8.5; Q6_K 6.6; Q5_K_M 5.7; Q4_K_M 4.9; Q3_K_M 3.9; Q2_K 3. Q4_K_M is about 4.9 GB rather than 4.0 because each block of weights carries a scale, and often a minimum, alongside the indices: roughly five bits per weight, not four. That extra gigabyte is the scales, and it is the answer to where it came from."}},"blocks":[{"t":"h2","id":"storing-a-number-in-four-bits","runs":[{"text":"Storing a number in four bits"}]},{"t":"p","runs":[{"text":"A weight in a released model is usually bfloat16: sixteen bits, of which eight are exponent. Quantisation stores it in fewer bits, and the whole art is in how you keep the information that matters."}]},{"t":"p","runs":[{"text":"Naively rounding every weight to one of sixteen values across the model’s full range would be a disaster, because a handful of large weights would stretch the range and everything else would collapse to zero. So quantisation is done "},{"text":"block-wise","b":true},{"text":". Split the weights into blocks of 32 or 64. For each block, find the largest magnitude, store a scale factor, and store each weight as a small integer index into that block’s range."}]},{"t":"code","lang":"","label":"Text","text":"block of 32 weights, 4-bit:\n  32 indices x 4 bits           = 128 bits\n  1 scale in fp16               =  16 bits\n  (some schemes also a minimum) =  16 bits\n  ------------------------------------------\n  160 bits / 32 weights          = 5.0 bits per weight"},{"t":"p","runs":[{"text":"This is why a model described as 4-bit is not exactly half the size of an 8-bit one. Real formats land between 4.5 and 5.5 bits per weight once the scales are counted, which is exactly why an 8B model at Q4_K_M is about 4.9 GB rather than 4.0 GB. If you have ever wondered where that extra gigabyte came from, it is the scales."}]},{"t":"h2","id":"the-k-in-k-quants","runs":[{"text":"The K in K-quants"}]},{"t":"p","runs":[{"text":"llama.cpp’s K-quant family does something smarter than uniform treatment: it assigns different precision to different tensors. The layers that hurt most when degraded (in practice, parts of the attention and one of the feed-forward matrices) keep more bits, while the rest take fewer. The suffixes encode this. "},{"text":"Q4_K_S","code":true},{"text":" is the small variant, "},{"text":"Q4_K_M","code":true},{"text":" the medium, and the M version spends its extra half-bit where measurements said it mattered."}]},{"t":"p","runs":[{"text":"More recent work adds an "},{"text":"importance matrix","b":true},{"text":": run a calibration corpus through the model, record which weights have the largest effect on the output, and let the quantiser protect them. Files named with "},{"text":"imatrix","code":true},{"text":" were built this way, and the benefit is largest at very low bit widths, where there is least room for error."}]},{"t":"h2","id":"the-other-methods-and-what-each-assumes","runs":[{"text":"The other methods, and what each assumes"}]},{"t":"p","runs":[{"text":"Round to nearest","b":true},{"text":" is the baseline: no calibration, instant, and the weakest."}]},{"t":"p","runs":[{"text":"GPTQ","b":true},{"text":" processes weights layer by layer, using a small calibration dataset and second-order information to compensate the remaining weights for each rounding error as it goes. Slow to produce, good results, GPU-oriented."}]},{"t":"p","runs":[{"text":"AWQ","b":true},{"text":" starts from an observation: roughly 1% of weight channels matter far more than the rest, and which ones can be identified from the "},{"text":"activations","i":true},{"text":" rather than the weights. Protect those channels by scaling, quantise the rest hard. Fast and strong, and widely used for GPU serving."}]},{"t":"p","runs":[{"text":"bitsandbytes NF4","b":true},{"text":" uses a 4-bit format shaped for normally distributed values, which weights approximately are. It is the format under QLoRA fine-tuning, and it is designed to be loaded and trained against rather than to be maximally compact."}]},{"t":"p","runs":[{"text":"FP8","b":true},{"text":" is not really quantisation in the same sense: it is a hardware-supported floating point format on recent Nvidia cards, giving near-lossless quality at half the memory with no calibration. Where the hardware supports it, it is the easy win."}]},{"t":"h2","id":"why-activation-outliers-matter","runs":[{"text":"Why activation outliers matter"}]},{"t":"p","runs":[{"text":"The reason naive 8-bit integer quantisation failed on transformers, and the reason the methods above exist, is a specific empirical fact: in large models a few hidden dimensions carry activations with magnitudes hundreds of times larger than the rest. Quantise a matrix multiplication that includes those channels uniformly and the small values disappear entirely."}]},{"t":"p","runs":[{"text":"Every serious method handles this somehow: by keeping outlier channels in higher precision, by scaling them before quantising, or by moving the difficulty from activations into weights. It is worth knowing because it explains why quantisation quality is not a smooth function of bit width and why methods that ignore the phenomenon fail suddenly rather than gradually."}]},{"t":"h2","id":"what-it-costs-in-one-sentence-you-can-act-on","runs":[{"text":"What it costs, in one sentence you can act on"}]},{"t":"p","runs":[{"text":"For models above about 7 billion parameters, going from 16-bit to a good 4-bit scheme typically costs a small fraction of a point on most tasks: usually invisible against the noise of a 40-example evaluation set."}]},{"t":"p","runs":[{"text":"Which produces the rule that decides most local setups: "},{"text":"a larger model quantised harder usually beats a smaller model at full precision","b":true},{"text":", for the same memory. A 13B at 4 bits and an 7B at 8 bits both occupy about 8 GB, and the 13B is generally better."}]},{"t":"p","runs":[{"text":"The rule has limits, and they matter. Below about 3 billion parameters the model has less redundancy to spare and quantisation hurts more. Quantisation degrades long-context retrieval and non-English performance more than it degrades short English chat, which is precisely the sort of thing a general benchmark hides. And below 4 bits the curve steepens sharply. Those limits are the subject of the next lesson."}]}],"check":{"question":[{"text":"Why is an 8B model in Q4_K_M about 4.9 GB rather than the 4.0 GB that four bits per weight would predict?"}],"options":[{"text":[{"text":"The tokenizer and configuration files add roughly a gigabyte to the download."}],"correct":false,"tempting":[{"text":"Those files are a few megabytes; they cannot account for a gigabyte."}]},{"text":[{"text":"Quantisation stores per-block scale factors alongside the indices, and K-quants keep selected tensors at higher precision, which together push the effective rate to about five bits per weight."}],"correct":true,"tempting":null},{"text":[{"text":"The embedding matrices cannot be quantised and remain at 16 bits."}],"correct":false,"tempting":[{"text":"Embeddings are often kept at higher precision than the average, but they are quantised too and are not the main cause."}]},{"text":[{"text":"GGUF files include the KV cache allocation in the file size."}],"correct":false,"tempting":[{"text":"The cache is allocated at runtime from your memory and is not stored in the model file at all."}]}],"order":[3,2,0,1],"why":null},"prev":{"slug":"choosing-for-the-job","title":"Choosing a Model for the Job","minutes":9},"next":{"slug":"choosing-a-bit-width","title":"Choosing a bit width you can defend","minutes":8}}}