Choosing a quantisation

The trade you are making

Weights are trained at 16-bit precision. Quantisation stores them with fewer bits: 8, 6, 5, 4, sometimes 3 or 2. The file shrinks roughly in proportion, generation speeds up because there is less memory to move, and quality degrades. The whole practical question is where on that curve to sit.

The shape of the curve is the useful thing to know, and it is not linear:

• 8-bit: indistinguishable from the original for practical purposes. Half the size.
• 6-bit: very close. A good choice when memory allows.
• 5-bit: small measurable degradation, rarely noticeable in use.
• 4-bit: the standard choice. Detectable on careful measurement, fine for everyday work. Quarter of the original size.
• 3-bit: noticeable. Instruction-following starts to slip.
• 2-bit: usually a false economy. A smaller model at 4-bit is almost always better than a large one at 2-bit.

That last line is the most useful rule in the lesson. Given a memory budget, prefer a smaller model at a comfortable quantisation over a larger model crushed to fit.

Reading the names

GGUF quantisations carry names like Q4_K_M. The parts: Q4 is 4-bit, _K means the newer k-quant scheme which is better than the old one at the same size, and _S, _M, _L are small, medium and large variants that keep more precision in the layers that matter most.

Q4_K_M is the default recommendation and has been for a long time. If you do not want to think about it, take that one. Q5_K_M if you have memory to spare, Q6_K if you have plenty.

You will also see IQ prefixes: importance-matrix quants, which use a calibration pass to decide which weights to keep precise. At the same file size they are generally better than the plain equivalent, particularly at the low end where 3-bit and 2-bit become viable rather than merely possible.

The other formats, and when they apply

GGUF: for llama.cpp, Ollama, LM Studio. Runs on CPU, GPU or both, on any platform. This is what you want on a laptop.

AWQ and GPTQ: 4-bit formats for GPU serving through vLLM and similar. Faster than GGUF on a proper GPU, no CPU fallback. This is what a hosting provider uses.

MLX: Apple’s format, for Apple Silicon. Noticeably faster than GGUF on a Mac for the same model.

bitsandbytes: quantise on the fly when loading a full-precision model in Python. Convenient for experiments, slower than a purpose-built format.

Judging it for yourself

Published perplexity numbers tell you a quantisation is 2% worse in a statistical sense. What you want to know is whether it is worse at your work, and the answer is often no.

Run four of your twelve prompts at Q4 and at Q6 and compare. On drafting, summarising and rewriting you will usually see no difference at all. On multi-step reasoning and code you sometimes will. That test takes fifteen minutes and settles it for your machine and your tasks.

One specific failure worth watching for, because it is the characteristic sign of quantising too far: the model starts ignoring parts of the instruction, or drifts out of the requested format halfway through a long answer. Facts degrade gently; instruction-following degrades noticeably.

The download itself

One practical note, because it catches people on limited connections. Model repositories hold every variant in the same place, and a naive download can fetch all of them: twenty files totalling 200 GB when you wanted one 5 GB file.

Fetch the specific quantisation by name. In Ollama, ollama pull qwen3:8b-instruct-q4_K_M names the variant directly. On Hugging Face, use the file browser and download the single .gguf you want, or pass a filename pattern to the command-line tool. Check the file size on the page before you start, and if you are on a metered connection, check it twice.

What quantisation does not do

It does not make a model know more. It does not extend the context window. It does not remove the licence terms. And it does not make an 8B model into a 70B: the ceiling on what a model can do is set by its training, and quantisation only decides how much of that survives on your hardware.