Quantisation, properly

"4-bit" does not mean every weight is rounded to one of sixteen values across the whole model. If it did, the model would be ruined. What actually happens is more careful, and understanding it tells you where the damage lands.

What 4-bit really means

Weights are quantised in blocks. A block of 32 weights is scaled so its values span the 4-bit range, and the 16-bit scale factor is stored alongside. Reconstruct with w = q * scale (plus an offset in some schemes).

So the real cost is 4 bits per weight plus 16 bits per 32 weights = 4.5 bits. The k-quant schemes in llama.cpp add a second level — super-blocks with their own scales — and spend more bits on the tensors that matter. Q4_K_M means: mostly 4-bit, but the attention value projections and the feed-forward down projections get 6-bit, and the embedding and output layers get more too. The measured average is about 4.8 bits per weight.

That mixed allocation is why a good 4-bit quant is close to the original and a naive one is not.

The four formats you will meet

GGUF (llama.cpp) is a file container plus a family of quant schemes. It runs on CPU, CUDA, Metal, Vulkan and ROCm, and supports partial offload — the only format that runs when the model does not fit. imatrix quants use an importance matrix computed over calibration text to decide which weights to protect; they are meaningfully better below 4 bits.

GPTQ quantises layer by layer, using second-order information about the layer's error to compensate as it goes. 4-bit with group size 128 is typical. GPU only, older, still widely supported by vLLM and TGI.

AWQ is activation-aware: it observes that roughly 1% of weight channels see large activations, and scales those channels up before quantising so they lose less. Usually 4-bit, group 128. Fast kernels, well supported in vLLM, and usually a little better than GPTQ at the same bit rate.

EXL2 (ExLlamaV2) assigns different bit rates to different tensors to hit a target average — you download a "4.65 bpw" model rather than a named tier. Calibrated, NVIDIA only, and the best quality per bit for a single user on a consumer card.

Also worth knowing: FP8 on Ada and Hopper cards is close to lossless and halves memory versus fp16, and Blackwell adds 4-bit floating point formats. Some newer models ship natively in 4-bit float rather than being quantised afterwards.

Where the quality actually goes

Quantisation damage does not look like damage. The model stays fluent. What degrades, roughly in order:

1. 1Following long instructions completely. Give it a fifteen-line spec and it quietly drops constraint number eleven.
2. 2Exact syntax. Code, JSON, regex, rare API names, correct escaping. A token that must be right has no room for approximation.
3. 3Low-resource languages and transliteration. These rely on the rarest parts of the weight distribution.
4. 4Arithmetic and multi-step chains, where a small error compounds.

Nothing on that list shows up in a casual chat. That is the trap.

Measuring it honestly

Perplexity deltas are what everyone publishes and they are close to useless at these scales — a change from 6.21 to 6.24 can hide a real drop in retrieval over long context. KL divergence against the fp16 model on the same inputs is a better signal, because it measures how differently the model *distributes* probability rather than how surprised it is.

The honest method is your own: thirty prompts from your real work, graded by you, fp16 versus the quant. It takes an hour and it answers the only question you have.

Rules of thumb worth having

• Q8 / 8-bit: effectively indistinguishable. Use it if it fits.
• Q6_K, Q5_K_M: safe for essentially everything.
• Q4_K_M / AWQ / GPTQ-4bit: the standard trade. Small, measurable, usually fine.
• Q3: noticeable on the four failure modes above.
• 2-bit: use a smaller model instead — unless the model is very large, where 70B at 2.4 bits still beats 8B at 8 bits. Large models tolerate crushing better.

And the general principle: given a fixed memory budget, more parameters at lower precision usually beats fewer parameters at higher precision, down to about 4 bits. Below that the advantage erodes.