K-quants and importance matrices

Rounding is not the hard part

Quantising a weight is trivial: divide by a scale, round, store the integer. The hard part is choosing the scales, and choosing which weights deserve protection. Everything that separates a good 4-bit model from a ruined one is in those two decisions.

Blocks, super-blocks and the real bit rate

A simple scheme takes 32 weights, finds the largest magnitude among them, sets a scale so the range fits, and stores 32 four-bit integers plus one 16-bit scale. That is 4 + 16/32 = 4.5 bits per weight, and the extra half bit is the reason a “4-bit” file is never exactly half the size of an 8-bit one.

K-quants add a level. Groups of blocks form a super-block with its own scale, and the per-block scales are themselves quantised to 6 bits against the super-block scale. The result stores finer-grained scales for less overhead, so a block whose weights happen to be tiny is not forced to share a scale with a block containing an outlier. Outliers are the whole problem: one large weight in a block stretches the scale and coarsens every other weight in it.

This is also why block size is a trade you can see. Smaller blocks track the local distribution better and cost more scale storage. Group size 128 in GPTQ and AWQ is the same choice made differently.

What an importance matrix does

An importance matrix, imatrix in llama.cpp, is computed by running the full-precision model over a body of text and recording, for each weight, how much the activations flowing through it contribute. Weights that are consistently multiplied by large activations matter more; an error there propagates. Weights that are rarely activated can be rounded harder.

The quantiser then uses that matrix to choose scales that minimise error weighted by importance, rather than raw error. The effect is small at 5 and 6 bits, clear at 4, and decisive below 4: the IQ2 and IQ3 schemes are essentially unusable without one.

Producing one:

llama-imatrix -m model-f16.gguf -f calibration.txt -o model.imatrix --chunks 200
llama-quantize --imatrix model.imatrix model-f16.gguf model-IQ3_M.gguf IQ3_M

It takes minutes to an hour depending on the model and the amount of text.

Calibration text is a choice with consequences

This is the part most people never think about and it is where a real trap lives.

The importance matrix reflects the text you fed it. Calibrate on English Wikipedia and the weights that matter for English prose are protected. The weights that matter for Bengali, for Python, for JSON escaping or for long-form instruction following were not exercised, so the quantiser was free to damage them, and it did, because that is what it was told to do.

The practical consequences:

• A general quant should be calibrated on diverse text (several languages, code, dialogue, structured output) and the well-known community quantisers do this deliberately.
• A quant for a specific job can be calibrated on that job’s text and be better than a general one at it, while being worse at everything else. If you run one narrow task at very low bit rates, this is a genuine and underused optimisation.
• Never calibrate on your evaluation set. You will measure a model tuned to the exact text you are testing on, and the number will be meaningless.

An honest limitation: the amount of calibration text used by most published quants is small (a few hundred thousand tokens is typical) which is enough to identify the broad importance structure and not enough to represent everything the model can do. This is one reason low-bit quants degrade unevenly across languages and tasks in ways that a single perplexity number does not reveal.

Choosing a tier, practically

• Q8_0: no imatrix needed, effectively lossless, half the size of fp16.
• Q6_K, Q5_K_M: safe for essentially everything. Imatrix optional.
• Q4_K_M: the standard trade. Imatrix helps slightly; take it if offered.
• IQ4_XS: about the same quality as Q4_K_M at a slightly smaller size, needs an imatrix, more compute to decode.
• Q3_K_M and IQ3: visible damage to precision-critical work. Imatrix now mandatory.
• IQ2: only for very large models where the alternative is not running them at all.

And the check that settles arguments: run your thirty-item audition set against two candidate quants. The difference between IQ4_XS and Q4_K_M is smaller than the difference between a good prompt and a bad one, and people spend far more time on the former.