{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"quantisation-properly","title":"Quantisation, properly","url":"https://addaly.com/learn/running-models-yourself/quantisation-properly","position":19,"of":82,"minutes":9,"answer":"Quantisation damages precision-critical work first (syntax, long instructions, rare tokens) while fluency stays intact.","keyIdea":"Quantisation damages precision-critical work first (syntax, long instructions, rare tokens) while fluency stays intact."},"body":{"format":"markdown","text":"\"4-bit\" does not mean every weight is rounded to one of sixteen values across the whole model. If it did, the model would be ruined. What actually happens is more careful, and understanding it tells you where the damage lands.\n\n## What 4-bit means\n\nWeights are quantised in **blocks**. A block of 32 weights is scaled so its values span the 4-bit range, and the 16-bit scale factor is stored alongside. Reconstruct with `w = q * scale` (plus an offset in some schemes).\n\nSo the real cost is 4 bits per weight plus 16 bits per 32 weights = 4.5 bits. The k-quant schemes in llama.cpp add a second level (super-blocks with their own scales) and spend more bits on the tensors that matter. `Q4_K_M` means: mostly 4-bit, but the attention value projections and the feed-forward down projections get 6-bit, and the embedding and output layers get more too. The measured average is about 4.8 bits per weight.\n\n> Figure: What a quantisation tier actually costs per weight. Bars (bits per weight, measured): fp16 or bf16 16; Q8_0 8.5; Q6_K 6.6; Q5_K_M 5.7; Q4_K_M 4.8; Q3_K_M 3.9; IQ2_XS 2.4. None of them is the number in the name. Each block of weights carries its own scale, so four nominal bits costs about 4.8 in practice, and a k-quant spends the difference deliberately, promoting the attention value projections, the feed-forward down projections and the output layer to six bits and more.\n\nThat mixed allocation is why a good 4-bit quant is close to the original and a naive one is not.\n\n## The four formats you will meet\n\n**GGUF** (llama.cpp) is a file container plus a family of quant schemes. It runs on CPU, CUDA, Metal, Vulkan and ROCm, and supports partial offload: the only format that runs when the model does not fit. `imatrix` quants use an importance matrix computed over calibration text to decide which weights to protect; they are meaningfully better below 4 bits.\n\n**GPTQ** quantises layer by layer, using second-order information about the layer's error to compensate as it goes. 4-bit with group size 128 is typical. GPU only, older, still widely supported by vLLM and TGI.\n\n**AWQ** is activation-aware: it observes that roughly 1% of weight channels see large activations, and scales those channels up before quantising so they lose less. Usually 4-bit, group 128. Fast kernels, well supported in vLLM, and usually a little better than GPTQ at the same bit rate.\n\n**EXL2** (ExLlamaV2) assigns different bit rates to different tensors to hit a target average: you download a \"4.65 bpw\" model rather than a named tier. Calibrated, NVIDIA only, and the best quality per bit for a single user on a consumer card.\n\nAlso worth knowing: **FP8** on Ada and Hopper cards is close to lossless and halves memory versus fp16, and Blackwell adds 4-bit floating point formats. Some newer models ship natively in 4-bit float rather than being quantised afterwards.\n\n## Where the quality actually goes\n\nQuantisation damage does not look like damage. The model stays fluent. What degrades, roughly in order:\n\n1. **Following long instructions completely.** Give it a fifteen-line spec and it drops constraint number eleven.\n2. **Exact syntax.** Code, JSON, regex, rare API names, correct escaping. A token that must be right has no room for approximation.\n3. **Low-resource languages and transliteration.** These rely on the rarest parts of the weight distribution.\n4. **Arithmetic and multi-step chains**, where a small error compounds.\n\nNothing on that list shows up in a casual chat. That is the trap.\n\n## Measuring it honestly\n\nPerplexity deltas are what everyone publishes and they are close to useless at these scales: a change from 6.21 to 6.24 can hide a real drop in retrieval over long context. KL divergence against the fp16 model on the same inputs is a better signal, because it measures how differently the model *distributes* probability rather than how surprised it is.\n\nThe honest method is your own: thirty prompts from your real work, graded by you, fp16 versus the quant. It takes an hour and it answers the only question you have.\n\n## Rules of thumb worth having\n\n- **Q8 / 8-bit**: effectively indistinguishable. Use it if it fits.\n- **Q6_K, Q5_K_M**: safe for essentially everything.\n- **Q4_K_M / AWQ / GPTQ-4bit**: the standard trade. Small, measurable, usually fine.\n- **Q3**: noticeable on the four failure modes above.\n- **2-bit**: use a smaller model instead, unless the model is very large, where 70B at 2.4 bits still beats 8B at 8 bits. Large models tolerate crushing better.\n\nAnd the general principle: given a fixed memory budget, more parameters at lower precision usually beats fewer parameters at higher precision, down to about 4 bits. Below that the advantage erodes."},"check":{"question":"You have about 8 GB of memory to spend on weights. You can run a 13B model at Q4_K_M (7.9 GB) or a 7B model at Q8_0 (7.2 GB). Which is the better default, and why?","options":[{"text":"The 13B at Q4_K_M, because losing precision costs less than halving parameter count, though verify on your own task if it involves exact syntax.","correct":true},{"text":"The 7B at Q8_0, because 8-bit is near-lossless and a lossless smaller model beats a damaged larger one.","correct":false,"tempting":"Treats precision loss as the dominant term when parameter count is doing more work at these bit rates."},{"text":"They will perform about the same, since the file sizes are nearly equal and size is what determines capability.","correct":false,"tempting":"Reads file size as a quality metric rather than a memory budget you are choosing how to spend."},{"text":"The 13B, and by the same logic a 70B at 2-bit would beat both, since parameters always win.","correct":false,"tempting":"Extends a rule that holds near 4 bits into a range where the trade genuinely reverses for smaller models."}]},"prev":{"slug":"when-it-does-not-fit","title":"When it does not fit: the ordered remedies","url":"https://addaly.com/learn/running-models-yourself/when-it-does-not-fit"},"next":{"slug":"inside-a-gguf-file","title":"Inside a GGUF file","url":"https://addaly.com/learn/running-models-yourself/inside-a-gguf-file"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Quantisation, properly', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/quantisation-properly (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Quantisation, properly","courseTitle":"Running Models Yourself","module":{"index":3,"slug":"quantisation-and-formats","title":"Quantisation and model formats"},"answer":null,"keyIdea":"Quantisation damages precision-critical work first (syntax, long instructions, rare tokens) while fluency stays intact.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 3.1","figure":{"type":"bars","unit":"bits per weight, measured","title":"What a quantisation tier actually costs per weight","series":[{"label":"fp16 or bf16","value":16},{"label":"Q8_0","value":8.5},{"label":"Q6_K","value":6.6},{"label":"Q5_K_M","value":5.7},{"label":"Q4_K_M","value":4.8,"highlight":true},{"label":"Q3_K_M","value":3.9},{"label":"IQ2_XS","value":2.4}],"caption":"None of them is the number in the name. Each block of weights carries its own scale, so four nominal bits costs about 4.8 in practice, and a k-quant spends the difference deliberately, promoting the attention value projections, the feed-forward down projections and the output layer to six bits and more."},"alt":"Figure: What a quantisation tier actually costs per weight. Bars (bits per weight, measured): fp16 or bf16 16; Q8_0 8.5; Q6_K 6.6; Q5_K_M 5.7; Q4_K_M 4.8; Q3_K_M 3.9; IQ2_XS 2.4. None of them is the number in the name. Each block of weights carries its own scale, so four nominal bits costs about 4.8 in practice, and a k-quant spends the difference deliberately, promoting the attention value projections, the feed-forward down projections and the output layer to six bits and more."}},"blocks":[{"t":"p","runs":[{"text":"“4-bit” does not mean every weight is rounded to one of sixteen values across the whole model. If it did, the model would be ruined. What actually happens is more careful, and understanding it tells you where the damage lands."}],"lede":true},{"t":"h2","id":"what-4-bit-means","runs":[{"text":"What 4-bit means"}]},{"t":"p","runs":[{"text":"Weights are quantised in "},{"text":"blocks","b":true},{"text":". A block of 32 weights is scaled so its values span the 4-bit range, and the 16-bit scale factor is stored alongside. Reconstruct with "},{"text":"w = q * scale","code":true},{"text":" (plus an offset in some schemes)."}]},{"t":"p","runs":[{"text":"So the real cost is 4 bits per weight plus 16 bits per 32 weights = 4.5 bits. The k-quant schemes in llama.cpp add a second level (super-blocks with their own scales) and spend more bits on the tensors that matter. "},{"text":"Q4_K_M","code":true},{"text":" means: mostly 4-bit, but the attention value projections and the feed-forward down projections get 6-bit, and the embedding and output layers get more too. The measured average is about 4.8 bits per weight."}]},{"t":"p","runs":[{"text":"That mixed allocation is why a good 4-bit quant is close to the original and a naive one is not."}]},{"t":"h2","id":"the-four-formats-you-will-meet","runs":[{"text":"The four formats you will meet"}]},{"t":"p","runs":[{"text":"GGUF","b":true},{"text":" (llama.cpp) is a file container plus a family of quant schemes. It runs on CPU, CUDA, Metal, Vulkan and ROCm, and supports partial offload: the only format that runs when the model does not fit. "},{"text":"imatrix","code":true},{"text":" quants use an importance matrix computed over calibration text to decide which weights to protect; they are meaningfully better below 4 bits."}]},{"t":"p","runs":[{"text":"GPTQ","b":true},{"text":" quantises layer by layer, using second-order information about the layer’s error to compensate as it goes. 4-bit with group size 128 is typical. GPU only, older, still widely supported by vLLM and TGI."}]},{"t":"p","runs":[{"text":"AWQ","b":true},{"text":" is activation-aware: it observes that roughly 1% of weight channels see large activations, and scales those channels up before quantising so they lose less. Usually 4-bit, group 128. Fast kernels, well supported in vLLM, and usually a little better than GPTQ at the same bit rate."}]},{"t":"p","runs":[{"text":"EXL2","b":true},{"text":" (ExLlamaV2) assigns different bit rates to different tensors to hit a target average: you download a “4.65 bpw” model rather than a named tier. Calibrated, NVIDIA only, and the best quality per bit for a single user on a consumer card."}]},{"t":"p","runs":[{"text":"Also worth knowing: "},{"text":"FP8","b":true},{"text":" on Ada and Hopper cards is close to lossless and halves memory versus fp16, and Blackwell adds 4-bit floating point formats. Some newer models ship natively in 4-bit float rather than being quantised afterwards."}]},{"t":"h2","id":"where-the-quality-actually-goes","runs":[{"text":"Where the quality actually goes"}]},{"t":"p","runs":[{"text":"Quantisation damage does not look like damage. The model stays fluent. What degrades, roughly in order:"}]},{"t":"ol","start":1,"items":[[{"text":"Following long instructions completely.","b":true},{"text":" Give it a fifteen-line spec and it drops constraint number eleven."}],[{"text":"Exact syntax.","b":true},{"text":" Code, JSON, regex, rare API names, correct escaping. A token that must be right has no room for approximation."}],[{"text":"Low-resource languages and transliteration.","b":true},{"text":" These rely on the rarest parts of the weight distribution."}],[{"text":"Arithmetic and multi-step chains","b":true},{"text":", where a small error compounds."}]]},{"t":"p","runs":[{"text":"Nothing on that list shows up in a casual chat. That is the trap."}]},{"t":"h2","id":"measuring-it-honestly","runs":[{"text":"Measuring it honestly"}]},{"t":"p","runs":[{"text":"Perplexity deltas are what everyone publishes and they are close to useless at these scales: a change from 6.21 to 6.24 can hide a real drop in retrieval over long context. KL divergence against the fp16 model on the same inputs is a better signal, because it measures how differently the model "},{"text":"distributes","i":true},{"text":" probability rather than how surprised it is."}]},{"t":"p","runs":[{"text":"The honest method is your own: thirty prompts from your real work, graded by you, fp16 versus the quant. It takes an hour and it answers the only question you have."}]},{"t":"h2","id":"rules-of-thumb-worth-having","runs":[{"text":"Rules of thumb worth having"}]},{"t":"ul","items":[[{"text":"Q8 / 8-bit","b":true},{"text":": effectively indistinguishable. Use it if it fits."}],[{"text":"Q6_K, Q5_K_M","b":true},{"text":": safe for essentially everything."}],[{"text":"Q4_K_M / AWQ / GPTQ-4bit","b":true},{"text":": the standard trade. Small, measurable, usually fine."}],[{"text":"Q3","b":true},{"text":": noticeable on the four failure modes above."}],[{"text":"2-bit","b":true},{"text":": use a smaller model instead, unless the model is very large, where 70B at 2.4 bits still beats 8B at 8 bits. Large models tolerate crushing better."}]]},{"t":"p","runs":[{"text":"And the general principle: given a fixed memory budget, more parameters at lower precision usually beats fewer parameters at higher precision, down to about 4 bits. Below that the advantage erodes."}]}],"check":{"question":[{"text":"You have about 8 GB of memory to spend on weights. You can run a 13B model at Q4_K_M (7.9 GB) or a 7B model at Q8_0 (7.2 GB). Which is the better default, and why?"}],"options":[{"text":[{"text":"The 13B at Q4_K_M, because losing precision costs less than halving parameter count, though verify on your own task if it involves exact syntax."}],"correct":true,"tempting":null},{"text":[{"text":"The 7B at Q8_0, because 8-bit is near-lossless and a lossless smaller model beats a damaged larger one."}],"correct":false,"tempting":[{"text":"Treats precision loss as the dominant term when parameter count is doing more work at these bit rates."}]},{"text":[{"text":"They will perform about the same, since the file sizes are nearly equal and size is what determines capability."}],"correct":false,"tempting":[{"text":"Reads file size as a quality metric rather than a memory budget you are choosing how to spend."}]},{"text":[{"text":"The 13B, and by the same logic a 70B at 2-bit would beat both, since parameters always win."}],"correct":false,"tempting":[{"text":"Extends a rule that holds near 4 bits into a range where the trade genuinely reverses for smaller models."}]}],"order":[1,2,0,3],"why":null},"prev":{"slug":"when-it-does-not-fit","title":"When it does not fit: the ordered remedies","minutes":9},"next":{"slug":"inside-a-gguf-file","title":"Inside a GGUF file","minutes":8}}}