{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"the-size-precision-frontier","title":"Bigger and crushed, or smaller and clean","url":"https://addaly.com/learn/running-models-yourself/the-size-precision-frontier","position":26,"of":82,"minutes":8,"answer":"For a fixed memory budget, more parameters at lower precision wins down to about 4 bits, but the rule optimises quality per gigabyte, and speed, context length and task specialisation routinely override it.","keyIdea":"For a fixed memory budget, more parameters at lower precision wins down to about 4 bits, but the rule optimises quality per gigabyte, and speed, context length and task specialisation routinely override it."},"body":{"format":"markdown","text":"## The question everybody has\n\nYou have 12 GB. You can run a 14B at roughly 4 bits, or an 8B at roughly 8 bits, or a 4B at full precision. All three fit. Which is best?\n\nThe general finding, repeated across many evaluations, is that **more parameters at lower precision wins, down to about 4 bits per weight**. A 14B at Q4_K_M beats an 8B at Q8_0 on most tasks, despite the 8B being nearly lossless and the 14B being visibly compressed.\n\nThe mechanism is worth understanding rather than memorising. A larger model has more redundancy: many weights encode overlapping information, so rounding any one of them loses less. It also knows and can do more, and the capability gap between size classes is larger than the gap quantisation opens within one.\n\n## Where the rule stops\n\nBelow about 4 bits the advantage erodes, and by 2 bits it usually reverses at small and medium sizes. Three reasons:\n\n1. **Quantisation error grows faster than linearly as bits fall.** Going from 8 to 4 bits costs little; from 4 to 3 costs noticeably more; from 3 to 2 costs a great deal more again.\n2. **Small models have less redundancy to spend.** A 7B has fewer spare parameters encoding the same thing, so each damaged weight matters more. Large models tolerate crushing far better, which is why a 70B at 2.4 bits remains useful while a 7B at 2 bits is not.\n3. **The scale overhead becomes significant.** At 2 nominal bits with per-block scales, the real rate is closer to 2.4–2.6, so you are paying a quarter of your budget on metadata.\n\nSo the practical ordering, for a fixed memory budget: prefer the largest model that fits at 4 bits or better; if reaching 4 bits requires a model so large that only 2-bit fits, step down a size class instead, unless that class is 70B or above.\n\n> Figure: Where bigger and crushed stops being the right answer. A two by two grid, rows A 7B model and A 70B model, columns At 4 bits and At 2 bits: A 7B model and At 4 bits: Take it; A 7B model and At 2 bits: Avoid; A 70B model and At 4 bits: Take it; A 70B model and At 2 bits: Usable. For a fixed memory budget, more parameters at lower precision wins down to about four bits per weight. Below that, quantisation error grows faster than linearly and a small model has less redundancy to spend, so each damaged weight matters more. That is why a 70B at 2.4 bits is still useful and a 7B at 2 bits is not.\n\n## What the rule ignores\n\nThree things make the rule wrong for specific people, and they are common enough to name.\n\n**Speed.** A 14B at Q4 reads about 8.5 GB per token; an 8B at Q4 reads 4.9. On a bandwidth-limited machine that is nearly a factor of two in tokens per second, and on a laptop the 14B may be unusably slow even though it fits. The rule optimises quality per gigabyte, not quality per second.\n\n**Context.** A bigger model leaves less room for the KV cache. If your work needs 32k of context, a 14B that fits at 4k and an 8B that fits at 32k are not competing on quality: one of them cannot do the job.\n\n**Task specialisation.** A 7B trained for code beats a general 14B at code, at half the size. Specialisation is a bigger lever than either parameter count or precision, and it is checked far less often.\n\n## A worked comparison\n\nSuppose 12 GB of VRAM, general assistant work, 8k context, about 1 GB overhead. That leaves roughly 10 GB for weights plus cache.\n\n- **14B at Q4_K_M**: 8.4 GB weights, about 0.8 GB cache. Fits. Best raw capability of the three.\n- **8B at Q6_K**: 6.6 GB weights, 1.0 GB cache. Fits easily, roughly 30% faster to generate, quality below the 14B on reasoning and instruction following.\n- **8B at Q4_K_M with 32k context**: 4.9 GB weights, 4.0 GB cache. Fits, and is the right answer if your documents are long.\n\nThere is no universal winner in that list, which is the honest conclusion. The rule chooses the first; your workload may choose the second or third, and it should.\n\n## How to settle it in practice\n\nDownload two candidates, not five. Run the audition set at temperature 0 against both, and time them. Then ask three questions: is the quality difference larger than a couple of items out of thirty; is the slower one still fast enough to use; and does either fail on context length. Those three answers decide it, and the whole exercise is an afternoon.\n\nThe reason to do it rather than follow the rule is that the rule was derived from benchmark averages across many tasks, and you have one task. Averages are a good prior and a poor conclusion."},"check":{"question":"On a laptop with 16 GB of unified memory and no discrete GPU, someone must choose between a 14B at Q4 and an 8B at Q5 for summarising documents of about 6,000 tokens. Both fit. What consideration most likely overrides the usual bigger-at-lower-precision rule?","options":[{"text":"Summarisation is size-sensitive, so the 14B's advantage will be larger here than the rule suggests.","correct":false,"tempting":"Inverts the sensitivity; summarising a document that is present in the prompt is among the least size-dependent tasks."},{"text":"Q5 quantisation preserves long-context recall better than Q4, which matters at 6,000 tokens.","correct":false,"tempting":"Attributes context behaviour to weight precision; recall over context is governed mainly by the model and the cache, not the weight bit rate."},{"text":"Decode is bandwidth-limited on this machine, so the 14B reads nearly twice the bytes per token and may be too slow to use on 6,000-token documents.","correct":true},{"text":"The 14B's KV cache at 6,000 tokens will not fit in 16 GB alongside the weights.","correct":false,"tempting":"Is arithmetically wrong (6,000 tokens of a modern GQA cache is well under a gigabyte) and answers a fit question that is not in doubt."}]},"prev":{"slug":"measuring-quantisation-damage","title":"Measuring the damage, properly","url":"https://addaly.com/learn/running-models-yourself/measuring-quantisation-damage"},"next":{"slug":"pruning-distillation-and-other-shrinking","title":"The other ways to make a model smaller","url":"https://addaly.com/learn/running-models-yourself/pruning-distillation-and-other-shrinking"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Bigger and crushed, or smaller and clean', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/the-size-precision-frontier (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Bigger and crushed, or smaller and clean","courseTitle":"Running Models Yourself","module":{"index":3,"slug":"quantisation-and-formats","title":"Quantisation and model formats"},"answer":null,"keyIdea":"For a fixed memory budget, more parameters at lower precision wins down to about 4 bits, but the rule optimises quality per gigabyte, and speed, context length and task specialisation routinely override it.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["At 4 bits","At 2 bits"],"rows":["A 7B model","A 70B model"],"type":"matrix","cells":[["Take it","Avoid"],["Take it","Usable"]],"title":"Where bigger and crushed stops being the right answer","caption":"For a fixed memory budget, more parameters at lower precision wins down to about four bits per weight. Below that, quantisation error grows faster than linearly and a small model has less redundancy to spend, so each damaged weight matters more. That is why a 70B at 2.4 bits is still useful and a 7B at 2 bits is not.","cellLabels":[["Loss is small","A 4B at 5 bits wins"],["Near the original","Beats an 8B at 8 bits"]]},"alt":"Figure: Where bigger and crushed stops being the right answer. A two by two grid, rows A 7B model and A 70B model, columns At 4 bits and At 2 bits: A 7B model and At 4 bits: Take it; A 7B model and At 2 bits: Avoid; A 70B model and At 4 bits: Take it; A 70B model and At 2 bits: Usable. For a fixed memory budget, more parameters at lower precision wins down to about four bits per weight. Below that, quantisation error grows faster than linearly and a small model has less redundancy to spend, so each damaged weight matters more. That is why a 70B at 2.4 bits is still useful and a 7B at 2 bits is not."}},"blocks":[{"t":"h2","id":"the-question-everybody-has","runs":[{"text":"The question everybody has"}]},{"t":"p","runs":[{"text":"You have 12 GB. You can run a 14B at roughly 4 bits, or an 8B at roughly 8 bits, or a 4B at full precision. All three fit. Which is best?"}]},{"t":"p","runs":[{"text":"The general finding, repeated across many evaluations, is that "},{"text":"more parameters at lower precision wins, down to about 4 bits per weight","b":true},{"text":". A 14B at Q4_K_M beats an 8B at Q8_0 on most tasks, despite the 8B being nearly lossless and the 14B being visibly compressed."}]},{"t":"p","runs":[{"text":"The mechanism is worth understanding rather than memorising. A larger model has more redundancy: many weights encode overlapping information, so rounding any one of them loses less. It also knows and can do more, and the capability gap between size classes is larger than the gap quantisation opens within one."}]},{"t":"h2","id":"where-the-rule-stops","runs":[{"text":"Where the rule stops"}]},{"t":"p","runs":[{"text":"Below about 4 bits the advantage erodes, and by 2 bits it usually reverses at small and medium sizes. Three reasons:"}]},{"t":"ol","start":1,"items":[[{"text":"Quantisation error grows faster than linearly as bits fall.","b":true},{"text":" Going from 8 to 4 bits costs little; from 4 to 3 costs noticeably more; from 3 to 2 costs a great deal more again."}],[{"text":"Small models have less redundancy to spend.","b":true},{"text":" A 7B has fewer spare parameters encoding the same thing, so each damaged weight matters more. Large models tolerate crushing far better, which is why a 70B at 2.4 bits remains useful while a 7B at 2 bits is not."}],[{"text":"The scale overhead becomes significant.","b":true},{"text":" At 2 nominal bits with per-block scales, the real rate is closer to 2.4–2.6, so you are paying a quarter of your budget on metadata."}]]},{"t":"p","runs":[{"text":"So the practical ordering, for a fixed memory budget: prefer the largest model that fits at 4 bits or better; if reaching 4 bits requires a model so large that only 2-bit fits, step down a size class instead, unless that class is 70B or above."}]},{"t":"h2","id":"what-the-rule-ignores","runs":[{"text":"What the rule ignores"}]},{"t":"p","runs":[{"text":"Three things make the rule wrong for specific people, and they are common enough to name."}]},{"t":"p","runs":[{"text":"Speed.","b":true},{"text":" A 14B at Q4 reads about 8.5 GB per token; an 8B at Q4 reads 4.9. On a bandwidth-limited machine that is nearly a factor of two in tokens per second, and on a laptop the 14B may be unusably slow even though it fits. The rule optimises quality per gigabyte, not quality per second."}]},{"t":"p","runs":[{"text":"Context.","b":true},{"text":" A bigger model leaves less room for the KV cache. If your work needs 32k of context, a 14B that fits at 4k and an 8B that fits at 32k are not competing on quality: one of them cannot do the job."}]},{"t":"p","runs":[{"text":"Task specialisation.","b":true},{"text":" A 7B trained for code beats a general 14B at code, at half the size. Specialisation is a bigger lever than either parameter count or precision, and it is checked far less often."}]},{"t":"h2","id":"a-worked-comparison","runs":[{"text":"A worked comparison"}]},{"t":"p","runs":[{"text":"Suppose 12 GB of VRAM, general assistant work, 8k context, about 1 GB overhead. That leaves roughly 10 GB for weights plus cache."}]},{"t":"ul","items":[[{"text":"14B at Q4_K_M","b":true},{"text":": 8.4 GB weights, about 0.8 GB cache. Fits. Best raw capability of the three."}],[{"text":"8B at Q6_K","b":true},{"text":": 6.6 GB weights, 1.0 GB cache. Fits easily, roughly 30% faster to generate, quality below the 14B on reasoning and instruction following."}],[{"text":"8B at Q4_K_M with 32k context","b":true},{"text":": 4.9 GB weights, 4.0 GB cache. Fits, and is the right answer if your documents are long."}]]},{"t":"p","runs":[{"text":"There is no universal winner in that list, which is the honest conclusion. The rule chooses the first; your workload may choose the second or third, and it should."}]},{"t":"h2","id":"how-to-settle-it-in-practice","runs":[{"text":"How to settle it in practice"}]},{"t":"p","runs":[{"text":"Download two candidates, not five. Run the audition set at temperature 0 against both, and time them. Then ask three questions: is the quality difference larger than a couple of items out of thirty; is the slower one still fast enough to use; and does either fail on context length. Those three answers decide it, and the whole exercise is an afternoon."}]},{"t":"p","runs":[{"text":"The reason to do it rather than follow the rule is that the rule was derived from benchmark averages across many tasks, and you have one task. Averages are a good prior and a poor conclusion."}]}],"check":{"question":[{"text":"On a laptop with 16 GB of unified memory and no discrete GPU, someone must choose between a 14B at Q4 and an 8B at Q5 for summarising documents of about 6,000 tokens. Both fit. What consideration most likely overrides the usual bigger-at-lower-precision rule?"}],"options":[{"text":[{"text":"Summarisation is size-sensitive, so the 14B’s advantage will be larger here than the rule suggests."}],"correct":false,"tempting":[{"text":"Inverts the sensitivity; summarising a document that is present in the prompt is among the least size-dependent tasks."}]},{"text":[{"text":"Q5 quantisation preserves long-context recall better than Q4, which matters at 6,000 tokens."}],"correct":false,"tempting":[{"text":"Attributes context behaviour to weight precision; recall over context is governed mainly by the model and the cache, not the weight bit rate."}]},{"text":[{"text":"Decode is bandwidth-limited on this machine, so the 14B reads nearly twice the bytes per token and may be too slow to use on 6,000-token documents."}],"correct":true,"tempting":null},{"text":[{"text":"The 14B’s KV cache at 6,000 tokens will not fit in 16 GB alongside the weights."}],"correct":false,"tempting":[{"text":"Is arithmetically wrong (6,000 tokens of a modern GQA cache is well under a gigabyte) and answers a fit question that is not in doubt."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"measuring-quantisation-damage","title":"Measuring the damage, properly","minutes":9},"next":{"slug":"pruning-distillation-and-other-shrinking","title":"The other ways to make a model smaller","minutes":8}}}