{"course":{"slug":"choosing-your-tools","title":"Choosing and Using the Tools","url":"https://addaly.com/learn/choosing-your-tools","level":"beginner","track":"work"},"lesson":{"slug":"quantisation-in-practice","title":"Choosing a quantisation","url":"https://addaly.com/learn/choosing-your-tools/quantisation-in-practice","position":43,"of":76,"minutes":8,"answer":"Take Q4_K_M unless you have memory to spare, prefer a smaller model at 4-bit over a larger one at 2-bit, and watch for drifting instruction-following as the sign that you have quantised too far.","keyIdea":"Take Q4_K_M unless you have memory to spare, prefer a smaller model at 4-bit over a larger one at 2-bit, and watch for drifting instruction-following as the sign that you have quantised too far."},"body":{"format":"markdown","text":"## The trade you are making\n\nWeights are trained at 16-bit precision. Quantisation stores them with fewer bits: 8, 6, 5, 4, sometimes 3 or 2. The file shrinks roughly in proportion, generation speeds up because there is less memory to move, and quality degrades. The whole practical question is where on that curve to sit.\n\nThe shape of the curve is the useful thing to know, and it is not linear:\n\n- **8-bit**: indistinguishable from the original for practical purposes. Half the size.\n- **6-bit**: very close. A good choice when memory allows.\n- **5-bit**: small measurable degradation, rarely noticeable in use.\n- **4-bit**: the standard choice. Detectable on careful measurement, fine for everyday work. Quarter of the original size.\n- **3-bit**: noticeable. Instruction-following starts to slip.\n- **2-bit**: usually a false economy. A smaller model at 4-bit is almost always better than a large one at 2-bit.\n\nThat last line is the most useful rule in the lesson. Given a memory budget, prefer a smaller model at a comfortable quantisation over a larger model crushed to fit.\n\n## Reading the names\n\nGGUF quantisations carry names like `Q4_K_M`. The parts: `Q4` is 4-bit, `_K` means the newer k-quant scheme which is better than the old one at the same size, and `_S`, `_M`, `_L` are small, medium and large variants that keep more precision in the layers that matter most.\n\n> Figure: The quantisation curve is not a straight line. Line chart of Everyday-task quality, % of the 16-bit model against Bits per weight: Instruct model on drafting, summarising and extraction (2, 71) (3, 88) (4, 97) (5, 98.5) (6, 99.4) (8, 99.9). Four bits is the standard choice because the curve is nearly flat above it and falls away sharply below. Given a fixed memory budget, a smaller model at four bits beats a larger one crushed to two, and the first sign that you have gone too far is instruction-following drifting rather than facts degrading.\n\n**`Q4_K_M` is the default recommendation** and has been for a long time. If you do not want to think about it, take that one. `Q5_K_M` if you have memory to spare, `Q6_K` if you have plenty.\n\nYou will also see `IQ` prefixes: importance-matrix quants, which use a calibration pass to decide which weights to keep precise. At the same file size they are generally better than the plain equivalent, particularly at the low end where 3-bit and 2-bit become viable rather than merely possible.\n\n## The other formats, and when they apply\n\n**GGUF**: for llama.cpp, Ollama, LM Studio. Runs on CPU, GPU or both, on any platform. This is what you want on a laptop.\n\n**AWQ** and **GPTQ**: 4-bit formats for GPU serving through vLLM and similar. Faster than GGUF on a proper GPU, no CPU fallback. This is what a hosting provider uses.\n\n**MLX**: Apple's format, for Apple Silicon. Noticeably faster than GGUF on a Mac for the same model.\n\n**bitsandbytes**: quantise on the fly when loading a full-precision model in Python. Convenient for experiments, slower than a purpose-built format.\n\n## Judging it for yourself\n\nPublished perplexity numbers tell you a quantisation is 2% worse in a statistical sense. What you want to know is whether it is worse at your work, and the answer is often no.\n\nRun four of your twelve prompts at Q4 and at Q6 and compare. On drafting, summarising and rewriting you will usually see no difference at all. On multi-step reasoning and code you sometimes will. That test takes fifteen minutes and settles it for your machine and your tasks.\n\nOne specific failure worth watching for, because it is the characteristic sign of quantising too far: the model starts ignoring parts of the instruction, or drifts out of the requested format halfway through a long answer. Facts degrade gently; instruction-following degrades noticeably.\n\n## The download itself\n\nOne practical note, because it catches people on limited connections. Model repositories hold every variant in the same place, and a naive download can fetch all of them: twenty files totalling 200 GB when you wanted one 5 GB file.\n\nFetch the specific quantisation by name. In Ollama, `ollama pull qwen3:8b-instruct-q4_K_M` names the variant directly. On Hugging Face, use the file browser and download the single `.gguf` you want, or pass a filename pattern to the command-line tool. Check the file size on the page before you start, and if you are on a metered connection, check it twice.\n\n## What quantisation does not do\n\nIt does not make a model know more. It does not extend the context window. It does not remove the licence terms. And it does not make an 8B model into a 70B: the ceiling on what a model can do is set by its training, and quantisation only decides how much of that survives on your hardware."},"check":{"question":"With 12 GB of memory, is a 32B model at 2-bit or a 14B model at 4-bit the better choice?","options":[{"text":"The 14B at 4-bit: degradation accelerates below about 3 bits and costs more than the parameters gain.","correct":true},{"text":"The 32B at 2-bit, since parameter count is the dominant factor in capability and quantisation only trims precision.","correct":false,"tempting":"Treats parameter count as decisive while ignoring that aggressive quantisation removes much of what those parameters encoded."},{"text":"Neither, since a 32B model cannot be loaded in 12 GB at any quantisation level.","correct":false,"tempting":"Asserts an impossibility that 2-bit quantisation specifically overcomes, which is why the choice arises at all."},{"text":"The 32B at 2-bit for knowledge questions and the 14B at 4-bit for anything requiring careful instruction-following.","correct":false,"tempting":"Splits the decision plausibly, but low-bit quantisation harms recall as well, so the split does not hold up in testing."}]},"prev":{"slug":"reading-a-model-card","title":"Reading a model card before you download","url":"https://addaly.com/learn/choosing-your-tools/reading-a-model-card"},"next":{"slug":"hosted-open-models","title":"Renting an open model instead of running it","url":"https://addaly.com/learn/choosing-your-tools/hosted-open-models"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Choosing a quantisation', in Choosing and Using the Tools. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/choosing-your-tools/quantisation-in-practice (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Choosing a quantisation","courseTitle":"Choosing and Using the Tools","module":{"index":5,"slug":"open-models-and-your-own-hardware","title":"Open models and your own hardware"},"answer":null,"keyIdea":"Take Q4_K_M unless you have memory to spare, prefer a smaller model at 4-bit over a larger one at 2-bit, and watch for drifting instruction-following as the sign that you have quantised too far.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"The quantisation curve is not a straight line","series":[{"label":"Instruct model on drafting, summarising and extraction","points":[[2,71],[3,88],[4,97],[5,98.5],[6,99.4],[8,99.9]]}],"xLabel":"Bits per weight","yLabel":"Everyday-task quality, % of the 16-bit model","caption":"Four bits is the standard choice because the curve is nearly flat above it and falls away sharply below. Given a fixed memory budget, a smaller model at four bits beats a larger one crushed to two, and the first sign that you have gone too far is instruction-following drifting rather than facts degrading."},"alt":"Figure: The quantisation curve is not a straight line. Line chart of Everyday-task quality, % of the 16-bit model against Bits per weight: Instruct model on drafting, summarising and extraction (2, 71) (3, 88) (4, 97) (5, 98.5) (6, 99.4) (8, 99.9). Four bits is the standard choice because the curve is nearly flat above it and falls away sharply below. Given a fixed memory budget, a smaller model at four bits beats a larger one crushed to two, and the first sign that you have gone too far is instruction-following drifting rather than facts degrading."}},"blocks":[{"t":"h2","id":"the-trade-you-are-making","runs":[{"text":"The trade you are making"}]},{"t":"p","runs":[{"text":"Weights are trained at 16-bit precision. Quantisation stores them with fewer bits: 8, 6, 5, 4, sometimes 3 or 2. The file shrinks roughly in proportion, generation speeds up because there is less memory to move, and quality degrades. The whole practical question is where on that curve to sit."}]},{"t":"p","runs":[{"text":"The shape of the curve is the useful thing to know, and it is not linear:"}]},{"t":"ul","items":[[{"text":"8-bit","b":true},{"text":": indistinguishable from the original for practical purposes. Half the size."}],[{"text":"6-bit","b":true},{"text":": very close. A good choice when memory allows."}],[{"text":"5-bit","b":true},{"text":": small measurable degradation, rarely noticeable in use."}],[{"text":"4-bit","b":true},{"text":": the standard choice. Detectable on careful measurement, fine for everyday work. Quarter of the original size."}],[{"text":"3-bit","b":true},{"text":": noticeable. Instruction-following starts to slip."}],[{"text":"2-bit","b":true},{"text":": usually a false economy. A smaller model at 4-bit is almost always better than a large one at 2-bit."}]]},{"t":"p","runs":[{"text":"That last line is the most useful rule in the lesson. Given a memory budget, prefer a smaller model at a comfortable quantisation over a larger model crushed to fit."}]},{"t":"h2","id":"reading-the-names","runs":[{"text":"Reading the names"}]},{"t":"p","runs":[{"text":"GGUF quantisations carry names like "},{"text":"Q4_K_M","code":true},{"text":". The parts: "},{"text":"Q4","code":true},{"text":" is 4-bit, "},{"text":"_K","code":true},{"text":" means the newer k-quant scheme which is better than the old one at the same size, and "},{"text":"_S","code":true},{"text":", "},{"text":"_M","code":true},{"text":", "},{"text":"_L","code":true},{"text":" are small, medium and large variants that keep more precision in the layers that matter most."}]},{"t":"p","runs":[{"text":"Q4_K_M","b":true,"code":true},{"text":" is the default recommendation","b":true},{"text":" and has been for a long time. If you do not want to think about it, take that one. "},{"text":"Q5_K_M","code":true},{"text":" if you have memory to spare, "},{"text":"Q6_K","code":true},{"text":" if you have plenty."}]},{"t":"p","runs":[{"text":"You will also see "},{"text":"IQ","code":true},{"text":" prefixes: importance-matrix quants, which use a calibration pass to decide which weights to keep precise. At the same file size they are generally better than the plain equivalent, particularly at the low end where 3-bit and 2-bit become viable rather than merely possible."}]},{"t":"h2","id":"the-other-formats-and-when-they-apply","runs":[{"text":"The other formats, and when they apply"}]},{"t":"p","runs":[{"text":"GGUF","b":true},{"text":": for llama.cpp, Ollama, LM Studio. Runs on CPU, GPU or both, on any platform. This is what you want on a laptop."}]},{"t":"p","runs":[{"text":"AWQ","b":true},{"text":" and "},{"text":"GPTQ","b":true},{"text":": 4-bit formats for GPU serving through vLLM and similar. Faster than GGUF on a proper GPU, no CPU fallback. This is what a hosting provider uses."}]},{"t":"p","runs":[{"text":"MLX","b":true},{"text":": Apple’s format, for Apple Silicon. Noticeably faster than GGUF on a Mac for the same model."}]},{"t":"p","runs":[{"text":"bitsandbytes","b":true},{"text":": quantise on the fly when loading a full-precision model in Python. Convenient for experiments, slower than a purpose-built format."}]},{"t":"h2","id":"judging-it-for-yourself","runs":[{"text":"Judging it for yourself"}]},{"t":"p","runs":[{"text":"Published perplexity numbers tell you a quantisation is 2% worse in a statistical sense. What you want to know is whether it is worse at your work, and the answer is often no."}]},{"t":"p","runs":[{"text":"Run four of your twelve prompts at Q4 and at Q6 and compare. On drafting, summarising and rewriting you will usually see no difference at all. On multi-step reasoning and code you sometimes will. That test takes fifteen minutes and settles it for your machine and your tasks."}]},{"t":"p","runs":[{"text":"One specific failure worth watching for, because it is the characteristic sign of quantising too far: the model starts ignoring parts of the instruction, or drifts out of the requested format halfway through a long answer. Facts degrade gently; instruction-following degrades noticeably."}]},{"t":"h2","id":"the-download-itself","runs":[{"text":"The download itself"}]},{"t":"p","runs":[{"text":"One practical note, because it catches people on limited connections. Model repositories hold every variant in the same place, and a naive download can fetch all of them: twenty files totalling 200 GB when you wanted one 5 GB file."}]},{"t":"p","runs":[{"text":"Fetch the specific quantisation by name. In Ollama, "},{"text":"ollama pull qwen3:8b-instruct-q4_K_M","code":true},{"text":" names the variant directly. On Hugging Face, use the file browser and download the single "},{"text":".gguf","code":true},{"text":" you want, or pass a filename pattern to the command-line tool. Check the file size on the page before you start, and if you are on a metered connection, check it twice."}]},{"t":"h2","id":"what-quantisation-does-not-do","runs":[{"text":"What quantisation does not do"}]},{"t":"p","runs":[{"text":"It does not make a model know more. It does not extend the context window. It does not remove the licence terms. And it does not make an 8B model into a 70B: the ceiling on what a model can do is set by its training, and quantisation only decides how much of that survives on your hardware."}]}],"check":{"question":[{"text":"With 12 GB of memory, is a 32B model at 2-bit or a 14B model at 4-bit the better choice?"}],"options":[{"text":[{"text":"The 14B at 4-bit: degradation accelerates below about 3 bits and costs more than the parameters gain."}],"correct":true,"tempting":null},{"text":[{"text":"The 32B at 2-bit, since parameter count is the dominant factor in capability and quantisation only trims precision."}],"correct":false,"tempting":[{"text":"Treats parameter count as decisive while ignoring that aggressive quantisation removes much of what those parameters encoded."}]},{"text":[{"text":"Neither, since a 32B model cannot be loaded in 12 GB at any quantisation level."}],"correct":false,"tempting":[{"text":"Asserts an impossibility that 2-bit quantisation specifically overcomes, which is why the choice arises at all."}]},{"text":[{"text":"The 32B at 2-bit for knowledge questions and the 14B at 4-bit for anything requiring careful instruction-following."}],"correct":false,"tempting":[{"text":"Splits the decision plausibly, but low-bit quantisation harms recall as well, so the split does not hold up in testing."}]}],"order":[2,3,1,0],"why":null},"prev":{"slug":"reading-a-model-card","title":"Reading a model card before you download","minutes":8},"next":{"slug":"hosted-open-models","title":"Renting an open model instead of running it","minutes":8}}}