Quantisation: what the numbers lose

What a quantised number actually is

A bf16 weight has sixteen bits: about three significant decimal digits and an enormous range. An int8 has 256 possible values. A four-bit integer has sixteen. Quantisation is the act of mapping each weight onto one of those few values in a way that loses as little as possible.

The simplest scheme stores one scale per group of weights. Take four weights: 0.31, −0.12, 0.05, 0.90. For signed four-bit integers the range is −8 to 7, so the scale is the largest magnitude divided by seven: 0.90 ÷ 7 ≈ 0.129. Each weight becomes round(w ÷ scale): 2, −1, 0, 7. To use them, multiply back: 0.26, −0.13, 0.00, 0.90.

The first weight has moved from 0.31 to 0.26, a 15 per cent error. That sounds unusable. It is not, because a weight is never used alone: it participates in a dot product over thousands of elements, and rounding errors in different directions largely cancel. The mechanism that makes quantisation work at all is averaging, which also predicts when it will fail: small models, with fewer weights per dot product, degrade more than large ones at the same bit width. They do.

Groups, and the outlier problem

If one scale covered a whole matrix, a single large value would ruin it. Suppose most weights are around 0.3 and one is 12. The scale becomes 12 ÷ 7 ≈ 1.7, and every ordinary weight rounds to zero. So scales are stored per group of 32 to 128 weights, in 16-bit. That is where the “4.5 bits per weight” figures come from: four bits of value and a share of a shared scale.

Weights are the easy half. In 2022 it was found that models above roughly six billion parameters develop a few hidden dimensions whose activations are twenty to a hundred times larger than the rest. Naive 8-bit activations collapsed on such models. The fix was to keep those few columns in 16-bit and quantise everything else: a mixed decomposition. This is why the common local recipe is four-bit weights with 16-bit activations, written W4A16, and why servers that quantise activations to 8-bit prefer the FP8 format, which has an exponent and copes with outliers where a plain integer cannot.

Better than rounding

Rounding each weight to its nearest level ignores which weights matter. Two methods do better, both free, both standard.

GPTQ quantises a matrix column by column, running a hundred or so calibration examples through the model, and after each column adjusts the not-yet-quantised weights to compensate for the error just introduced. The compensation uses second-order information about the loss, and the result is a four-bit model markedly closer to the original than rounding gives.

AWQ observes that a small fraction of weight channels (around one per cent) carry most of the signal, identifiable from activation magnitudes on calibration data. It scales those channels up before quantising so they receive more of the available resolution, and folds the scaling into the next layer.

In the GGUF world that llama.cpp and Ollama use, the names encode the recipe. Q8_0 is 8-bit and effectively lossless. Q4_K_M is four-bit with grouped scales in a superblock layout, with a few sensitive tensors (attention values, the output projection) kept at higher precision; it is the default trade almost everyone lands on. Q2 and Q3 variants visibly degrade. Files marked IQ or built with an “importance matrix” used calibration data in the AWQ spirit.

What degrades, and where it hides

Measured on perplexity (module five’s loss, exponentiated) an 8-billion model at Q4_K_M is typically one to three per cent worse than bf16, and at 8-bit it is indistinguishable. That number is quoted everywhere and it hides the damage.

Perplexity averages over every token of ordinary text, most of which is easy. The loss from quantisation concentrates where the margins were already thin:

• Multi-step arithmetic and code, where small per-step errors compound across steps. A model two per cent worse on perplexity can be ten points worse on a maths benchmark at four bits.
• Long-context recall, especially if the cache is quantised too. 8-bit cache is close to free; 4-bit cache costs measurable needle-finding accuracy.
• Low-resource languages. Module six showed those directions were weakly represented to begin with; they have the least margin to lose.
• Precise instruction-following (formatting constraints, exact lengths) which lives in the same thin margins.

Larger models tolerate low bit widths better, for the averaging reason. A four-bit 70-billion model is usually excellent. A four-bit 1-billion model is often noticeably damaged, and 3-bit versions of small models are frequently not worth running.

The quantisation you cannot see

Providers quantise too. FP8 serving on current data-centre hardware is routine, and serving-side precision is one of the things that can change under a fixed model name. It is one of the signatures in module six’s drift lesson: a small, broad degradation, worst on arithmetic and long-context tasks, appearing on a date.

The test that replaces the perplexity number

Do not choose a quantisation level from a chart. Take your own hundred-item evaluation set, run it at bf16 and at the candidate quantisation, and compare: per task, not overall. If the drop sits inside the noise band you measured in module six, take the speed and the memory. If it does not, try the next level up, or a bigger model at the lower precision.

That last comparison matters more than it looks. A four-bit 14-billion model usually beats a bf16 7-billion model on quality and costs about the same memory. A 2-bit 70-billion model often loses to a four-bit 32-billion model. Quantisation is one axis of a two-axis trade with model size, and the right point is found by measuring, in an afternoon, with free tools.