What quantisation actually does

Storing a number in four bits

A weight in a released model is usually bfloat16: sixteen bits, of which eight are exponent. Quantisation stores it in fewer bits, and the whole art is in how you keep the information that matters.

Naively rounding every weight to one of sixteen values across the model’s full range would be a disaster, because a handful of large weights would stretch the range and everything else would collapse to zero. So quantisation is done block-wise. Split the weights into blocks of 32 or 64. For each block, find the largest magnitude, store a scale factor, and store each weight as a small integer index into that block’s range.

block of 32 weights, 4-bit:
  32 indices x 4 bits           = 128 bits
  1 scale in fp16               =  16 bits
  (some schemes also a minimum) =  16 bits
  ------------------------------------------
  160 bits / 32 weights          = 5.0 bits per weight

This is why a model described as 4-bit is not exactly half the size of an 8-bit one. Real formats land between 4.5 and 5.5 bits per weight once the scales are counted, which is exactly why an 8B model at Q4_K_M is about 4.9 GB rather than 4.0 GB. If you have ever wondered where that extra gigabyte came from, it is the scales.

The K in K-quants

llama.cpp’s K-quant family does something smarter than uniform treatment: it assigns different precision to different tensors. The layers that hurt most when degraded (in practice, parts of the attention and one of the feed-forward matrices) keep more bits, while the rest take fewer. The suffixes encode this. Q4_K_S is the small variant, Q4_K_M the medium, and the M version spends its extra half-bit where measurements said it mattered.

More recent work adds an importance matrix: run a calibration corpus through the model, record which weights have the largest effect on the output, and let the quantiser protect them. Files named with imatrix were built this way, and the benefit is largest at very low bit widths, where there is least room for error.

The other methods, and what each assumes

Round to nearest is the baseline: no calibration, instant, and the weakest.

GPTQ processes weights layer by layer, using a small calibration dataset and second-order information to compensate the remaining weights for each rounding error as it goes. Slow to produce, good results, GPU-oriented.

AWQ starts from an observation: roughly 1% of weight channels matter far more than the rest, and which ones can be identified from the activations rather than the weights. Protect those channels by scaling, quantise the rest hard. Fast and strong, and widely used for GPU serving.

bitsandbytes NF4 uses a 4-bit format shaped for normally distributed values, which weights approximately are. It is the format under QLoRA fine-tuning, and it is designed to be loaded and trained against rather than to be maximally compact.

FP8 is not really quantisation in the same sense: it is a hardware-supported floating point format on recent Nvidia cards, giving near-lossless quality at half the memory with no calibration. Where the hardware supports it, it is the easy win.

Why activation outliers matter

The reason naive 8-bit integer quantisation failed on transformers, and the reason the methods above exist, is a specific empirical fact: in large models a few hidden dimensions carry activations with magnitudes hundreds of times larger than the rest. Quantise a matrix multiplication that includes those channels uniformly and the small values disappear entirely.

Every serious method handles this somehow: by keeping outlier channels in higher precision, by scaling them before quantising, or by moving the difficulty from activations into weights. It is worth knowing because it explains why quantisation quality is not a smooth function of bit width and why methods that ignore the phenomenon fail suddenly rather than gradually.

What it costs, in one sentence you can act on

For models above about 7 billion parameters, going from 16-bit to a good 4-bit scheme typically costs a small fraction of a point on most tasks: usually invisible against the noise of a 40-example evaluation set.

Which produces the rule that decides most local setups: a larger model quantised harder usually beats a smaller model at full precision, for the same memory. A 13B at 4 bits and an 7B at 8 bits both occupy about 8 GB, and the 13B is generally better.

The rule has limits, and they matter. Below about 3 billion parameters the model has less redundancy to spare and quantisation hurts more. Quantisation degrades long-context retrieval and non-English performance more than it degrades short English chat, which is precisely the sort of thing a general benchmark hides. And below 4 bits the curve steepens sharply. Those limits are the subject of the next lesson.