{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"choosing-a-bit-width","title":"Choosing a bit width you can defend","url":"https://addaly.com/learn/open-models/choosing-a-bit-width","position":33,"of":71,"minutes":8,"answer":"Spend memory on parameters before precision down to about four bits, but expect quantisation to damage long-context retrieval and non-English performance well before it damages short English chat.","keyIdea":"Spend memory on parameters before precision down to about four bits, but expect quantisation to damage long-context retrieval and non-English performance well before it damages short English chat, so test at the bit width you will ship."},"body":{"format":"markdown","text":"## The ladder, and what each rung costs\n\nOrdered from least to most compressed, with the honest character of each:\n\n- **bf16 / fp16**: the reference. Everything else is measured against this.\n- **fp8 or Q8_0**: effectively indistinguishable from the reference on ordinary tasks. Use it when memory allows; there is nothing to think about.\n- **Q6_K**: very close to Q8 at three-quarters of the size. An underused sweet spot for people with a little headroom.\n- **Q5_K_M**: a small, usually unnoticed loss. Sensible when Q4 makes you nervous and Q6 does not fit.\n- **Q4_K_M**: the community default, and correctly so. The knee of the curve for models above about 7B.\n- **Q3_K_M**: noticeably degraded. Instruction adherence weakens first; the model starts ignoring parts of a multi-clause request.\n- **Q2_K**: usually a mistake. A smaller model at Q4 will almost always be better.\n\n## The rule that follows\n\n**Spend your memory on parameters before you spend it on precision, down to about 4 bits.**\n\nGiven 8 GB to work with, a 13B at Q4 beats an 8B at Q6 beats a 7B at Q8, on most tasks, most of the time. Given 5 GB, an 8B at Q4 beats a 4B at Q8.\n\nThe rule inverts below 4 bits and at small sizes, and both exceptions have the same cause. Quantisation works because large models carry redundancy: many weights contribute a little, so small errors average out. A 1.5B model has less of that redundancy per parameter, so the same bit reduction costs it more. Below about 3B, prefer Q6 or Q8 and accept the smaller model.\n\n## What degrades first, and where the benchmarks hide it\n\nThe damage is not spread evenly across capabilities, and a general benchmark average conceals this. In rough order of sensitivity:\n\n1. **Long-context retrieval.** Finding a specific fact deep in a long document depends on fine distinctions between key vectors, which is exactly what coarse quantisation blurs.\n2. **Non-English performance.** Lower-resource languages sit in less well-populated regions of the model's representation, with less redundancy supporting them. A quantised model can lose noticeably more of its Hindi than its English.\n3. **Multi-step arithmetic and code.** Errors compound; a small increase in per-step error is visible at the end of a chain.\n4. **Instruction adherence over long outputs.** The model complies for two paragraphs and drifts.\n5. **Short English chat.** The last thing to break, and unfortunately the first thing everyone tests.\n\nIf any of the first four describe your use, test at your quantisation rather than trusting a published perplexity number.\n\n## Perplexity is a weak proxy\n\nMost quantisation comparisons report perplexity on a text corpus. It is cheap, it correlates with quality, and it is not sensitive enough for this decision: two quantisations can be within a hundredth of a perplexity point and behave differently on structured output.\n\nThe better measurement, covered in more detail later, is agreement with the full-precision model. Run both on 200 prompts and compare their next-token distributions, or more compare their actual answers on your own examples. A quantisation that agrees with the reference on 96% of your task is a different proposition from one that agrees on 78%, and perplexity may not separate them.\n\n## A practical procedure\n\n1. Compute how much memory you have for weights, after subtracting KV cache and about 1 to 2 GB of overhead.\n2. Pick the **largest model** whose Q4_K_M fits.\n3. If it fits with room to spare, move up to Q5 or Q6 rather than to a bigger model, unless the next size up also fits at Q4.\n4. Run your 40 examples at the chosen quantisation and at one step higher.\n5. If the difference is inside the noise, keep the smaller file. If it is not, you have learned something specific to your task that no general table would have told you.\n\nStep 4 costs an hour and settles arguments that otherwise recur for months.\n\n> On downloading: repositories from prolific quantisers such as bartowski and the official organisation uploads generally publish the whole ladder with the importance-matrix variants. Take Q4_K_M first, and take one neighbour so you can run step 4 without a second download."},"check":{"question":"A team runs a Q4_K_M 8B model for Bengali summarisation and finds quality clearly worse than the fp16 version, although published perplexity and English benchmark differences were negligible. What is the most likely reason?","options":[{"text":"Quantisation is unreliable for any model below 13B, so the 8B is simply too small to quantise.","correct":false,"tempting":"The size sensitivity is real below roughly 3B; an 8B quantises well on English, which is why the language is the distinguishing factor here."},{"text":"The GGUF conversion lost the Bengali tokens from the vocabulary.","correct":false,"tempting":"Conversion can damage templates and stop tokens, but the vocabulary is copied wholesale and a missing token set would break generation entirely."},{"text":"Lower-resource languages rely on less redundant regions of the model, so they lose more to quantisation than English does, and English-weighted benchmarks do not show it.","correct":true},{"text":"Perplexity was measured on the wrong corpus, so the published numbers are invalid.","correct":false,"tempting":"Perplexity is genuinely a weak proxy here, but the corpus is not the core issue: the damage is unevenly distributed across capabilities."}]},"prev":{"slug":"quantisation-explained","title":"What quantisation actually does","url":"https://addaly.com/learn/open-models/quantisation-explained"},"next":{"slug":"llama-cpp-in-practice","title":"llama.cpp, end to end","url":"https://addaly.com/learn/open-models/llama-cpp-in-practice"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Choosing a bit width you can defend', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/choosing-a-bit-width (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Choosing a bit width you can defend","courseTitle":"The Open Model Ecosystem","module":{"index":4,"slug":"making-it-run","title":"Making it run on the machine you have"},"answer":null,"keyIdea":"Spend memory on parameters before precision down to about four bits, but expect quantisation to damage long-context retrieval and non-English performance well before it damages short English chat, so test at the bit width you will ship.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-ladder-and-what-each-rung-costs","runs":[{"text":"The ladder, and what each rung costs"}]},{"t":"p","runs":[{"text":"Ordered from least to most compressed, with the honest character of each:"}]},{"t":"ul","items":[[{"text":"bf16 / fp16","b":true},{"text":": the reference. Everything else is measured against this."}],[{"text":"fp8 or Q8_0","b":true},{"text":": effectively indistinguishable from the reference on ordinary tasks. Use it when memory allows; there is nothing to think about."}],[{"text":"Q6_K","b":true},{"text":": very close to Q8 at three-quarters of the size. An underused sweet spot for people with a little headroom."}],[{"text":"Q5_K_M","b":true},{"text":": a small, usually unnoticed loss. Sensible when Q4 makes you nervous and Q6 does not fit."}],[{"text":"Q4_K_M","b":true},{"text":": the community default, and correctly so. The knee of the curve for models above about 7B."}],[{"text":"Q3_K_M","b":true},{"text":": noticeably degraded. Instruction adherence weakens first; the model starts ignoring parts of a multi-clause request."}],[{"text":"Q2_K","b":true},{"text":": usually a mistake. A smaller model at Q4 will almost always be better."}]]},{"t":"h2","id":"the-rule-that-follows","runs":[{"text":"The rule that follows"}]},{"t":"p","runs":[{"text":"Spend your memory on parameters before you spend it on precision, down to about 4 bits.","b":true}]},{"t":"p","runs":[{"text":"Given 8 GB to work with, a 13B at Q4 beats an 8B at Q6 beats a 7B at Q8, on most tasks, most of the time. Given 5 GB, an 8B at Q4 beats a 4B at Q8."}]},{"t":"p","runs":[{"text":"The rule inverts below 4 bits and at small sizes, and both exceptions have the same cause. Quantisation works because large models carry redundancy: many weights contribute a little, so small errors average out. A 1.5B model has less of that redundancy per parameter, so the same bit reduction costs it more. Below about 3B, prefer Q6 or Q8 and accept the smaller model."}]},{"t":"h2","id":"what-degrades-first-and-where-the-benchmarks-hide-it","runs":[{"text":"What degrades first, and where the benchmarks hide it"}]},{"t":"p","runs":[{"text":"The damage is not spread evenly across capabilities, and a general benchmark average conceals this. In rough order of sensitivity:"}]},{"t":"ol","start":1,"items":[[{"text":"Long-context retrieval.","b":true},{"text":" Finding a specific fact deep in a long document depends on fine distinctions between key vectors, which is exactly what coarse quantisation blurs."}],[{"text":"Non-English performance.","b":true},{"text":" Lower-resource languages sit in less well-populated regions of the model’s representation, with less redundancy supporting them. A quantised model can lose noticeably more of its Hindi than its English."}],[{"text":"Multi-step arithmetic and code.","b":true},{"text":" Errors compound; a small increase in per-step error is visible at the end of a chain."}],[{"text":"Instruction adherence over long outputs.","b":true},{"text":" The model complies for two paragraphs and drifts."}],[{"text":"Short English chat.","b":true},{"text":" The last thing to break, and unfortunately the first thing everyone tests."}]]},{"t":"p","runs":[{"text":"If any of the first four describe your use, test at your quantisation rather than trusting a published perplexity number."}]},{"t":"h2","id":"perplexity-is-a-weak-proxy","runs":[{"text":"Perplexity is a weak proxy"}]},{"t":"p","runs":[{"text":"Most quantisation comparisons report perplexity on a text corpus. It is cheap, it correlates with quality, and it is not sensitive enough for this decision: two quantisations can be within a hundredth of a perplexity point and behave differently on structured output."}]},{"t":"p","runs":[{"text":"The better measurement, covered in more detail later, is agreement with the full-precision model. Run both on 200 prompts and compare their next-token distributions, or more compare their actual answers on your own examples. A quantisation that agrees with the reference on 96% of your task is a different proposition from one that agrees on 78%, and perplexity may not separate them."}]},{"t":"h2","id":"a-practical-procedure","runs":[{"text":"A practical procedure"}]},{"t":"ol","start":1,"items":[[{"text":"Compute how much memory you have for weights, after subtracting KV cache and about 1 to 2 GB of overhead."}],[{"text":"Pick the "},{"text":"largest model","b":true},{"text":" whose Q4_K_M fits."}],[{"text":"If it fits with room to spare, move up to Q5 or Q6 rather than to a bigger model, unless the next size up also fits at Q4."}],[{"text":"Run your 40 examples at the chosen quantisation and at one step higher."}],[{"text":"If the difference is inside the noise, keep the smaller file. If it is not, you have learned something specific to your task that no general table would have told you."}]]},{"t":"p","runs":[{"text":"Step 4 costs an hour and settles arguments that otherwise recur for months."}]},{"t":"quote","runs":[{"text":"On downloading: repositories from prolific quantisers such as bartowski and the official organisation uploads generally publish the whole ladder with the importance-matrix variants. Take Q4_K_M first, and take one neighbour so you can run step 4 without a second download."}]}],"check":{"question":[{"text":"A team runs a Q4_K_M 8B model for Bengali summarisation and finds quality clearly worse than the fp16 version, although published perplexity and English benchmark differences were negligible. What is the most likely reason?"}],"options":[{"text":[{"text":"Quantisation is unreliable for any model below 13B, so the 8B is simply too small to quantise."}],"correct":false,"tempting":[{"text":"The size sensitivity is real below roughly 3B; an 8B quantises well on English, which is why the language is the distinguishing factor here."}]},{"text":[{"text":"The GGUF conversion lost the Bengali tokens from the vocabulary."}],"correct":false,"tempting":[{"text":"Conversion can damage templates and stop tokens, but the vocabulary is copied wholesale and a missing token set would break generation entirely."}]},{"text":[{"text":"Lower-resource languages rely on less redundant regions of the model, so they lose more to quantisation than English does, and English-weighted benchmarks do not show it."}],"correct":true,"tempting":null},{"text":[{"text":"Perplexity was measured on the wrong corpus, so the published numbers are invalid."}],"correct":false,"tempting":[{"text":"Perplexity is genuinely a weak proxy here, but the corpus is not the core issue: the damage is unevenly distributed across capabilities."}]}],"order":[0,1,3,2],"why":null},"prev":{"slug":"quantisation-explained","title":"What quantisation actually does","minutes":10},"next":{"slug":"llama-cpp-in-practice","title":"llama.cpp, end to end","minutes":10}}}