{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"k-quants-and-importance-matrices","title":"K-quants and importance matrices","url":"https://addaly.com/learn/running-models-yourself/k-quants-and-importance-matrices","position":21,"of":82,"minutes":9,"answer":"K-quants spend bits where the model is sensitive and importance matrices decide where that is, so the calibration text silently determines which languages and tasks a low-bit quant preserves.","keyIdea":"K-quants spend bits where the model is sensitive and importance matrices decide where that is, so the calibration text silently determines which languages and tasks a low-bit quant preserves."},"body":{"format":"markdown","text":"## Rounding is not the hard part\n\nQuantising a weight is trivial: divide by a scale, round, store the integer. The hard part is choosing the scales, and choosing which weights deserve protection. Everything that separates a good 4-bit model from a ruined one is in those two decisions.\n\n## Blocks, super-blocks and the real bit rate\n\nA simple scheme takes 32 weights, finds the largest magnitude among them, sets a scale so the range fits, and stores 32 four-bit integers plus one 16-bit scale. That is 4 + 16/32 = 4.5 bits per weight, and the extra half bit is the reason a \"4-bit\" file is never exactly half the size of an 8-bit one.\n\nK-quants add a level. Groups of blocks form a super-block with its own scale, and the per-block scales are themselves quantised to 6 bits against the super-block scale. The result stores finer-grained scales for less overhead, so a block whose weights happen to be tiny is not forced to share a scale with a block containing an outlier. Outliers are the whole problem: one large weight in a block stretches the scale and coarsens every other weight in it.\n\nThis is also why block size is a trade you can see. Smaller blocks track the local distribution better and cost more scale storage. Group size 128 in GPTQ and AWQ is the same choice made differently.\n\n## What an importance matrix does\n\nAn importance matrix, `imatrix` in llama.cpp, is computed by running the full-precision model over a body of text and recording, for each weight, how much the activations flowing through it contribute. Weights that are consistently multiplied by large activations matter more; an error there propagates. Weights that are rarely activated can be rounded harder.\n\nThe quantiser then uses that matrix to choose scales that minimise error weighted by importance, rather than raw error. The effect is small at 5 and 6 bits, clear at 4, and decisive below 4: the IQ2 and IQ3 schemes are essentially unusable without one.\n\nProducing one:\n\n```bash\nllama-imatrix -m model-f16.gguf -f calibration.txt -o model.imatrix --chunks 200\nllama-quantize --imatrix model.imatrix model-f16.gguf model-IQ3_M.gguf IQ3_M\n```\n\nIt takes minutes to an hour depending on the model and the amount of text.\n\n## Calibration text is a choice with consequences\n\nThis is the part most people never think about and it is where a real trap lives.\n\nThe importance matrix reflects the text you fed it. Calibrate on English Wikipedia and the weights that matter for English prose are protected. The weights that matter for Bengali, for Python, for JSON escaping or for long-form instruction following were not exercised, so the quantiser was free to damage them, and it did, because that is what it was told to do.\n\nThe practical consequences:\n\n- **A general quant should be calibrated on diverse text** (several languages, code, dialogue, structured output) and the well-known community quantisers do this deliberately.\n- **A quant for a specific job can be calibrated on that job's text and be better than a general one at it**, while being worse at everything else. If you run one narrow task at very low bit rates, this is a genuine and underused optimisation.\n- **Never calibrate on your evaluation set.** You will measure a model tuned to the exact text you are testing on, and the number will be meaningless.\n\nAn honest limitation: the amount of calibration text used by most published quants is small (a few hundred thousand tokens is typical) which is enough to identify the broad importance structure and not enough to represent everything the model can do. This is one reason low-bit quants degrade unevenly across languages and tasks in ways that a single perplexity number does not reveal.\n\n## Choosing a tier, practically\n\n- **Q8_0**: no imatrix needed, effectively lossless, half the size of fp16.\n- **Q6_K, Q5_K_M**: safe for essentially everything. Imatrix optional.\n- **Q4_K_M**: the standard trade. Imatrix helps slightly; take it if offered.\n- **IQ4_XS**: about the same quality as Q4_K_M at a slightly smaller size, needs an imatrix, more compute to decode.\n- **Q3_K_M and IQ3**: visible damage to precision-critical work. Imatrix now mandatory.\n- **IQ2**: only for very large models where the alternative is not running them at all.\n\nAnd the check that settles arguments: run your thirty-item audition set against two candidate quants. The difference between IQ4_XS and Q4_K_M is smaller than the difference between a good prompt and a bad one, and people spend far more time on the former."},"check":{"question":"A team quantises a multilingual model to IQ3_M using an importance matrix calibrated on English technical documentation. English output is fine; Hindi output degrades sharply compared with the Q5_K_M version. What is the mechanism?","options":[{"text":"Devanagari tokens occupy more of the vocabulary, so the output layer is under greater pressure at low bit rates regardless of calibration.","correct":false,"tempting":"Identifies a genuine tokenisation cost that affects speed and context use, not a mechanism by which calibration language changes weight protection."},{"text":"IQ schemes are known to be unsuitable for non-Latin scripts because their codebooks are derived from Latin-script statistics.","correct":false,"tempting":"Invents a property of the format; the lattice codebooks are script-agnostic and it is the imatrix, not the scheme, that carries language bias."},{"text":"The importance matrix recorded which weights matter for English text, so the weights carrying Hindi capability were treated as unimportant and rounded hardest.","correct":true},{"text":"Three-bit quantisation cannot represent multilingual embeddings at all, so any IQ3 build will lose non-English languages.","correct":false,"tempting":"Overstates a gradual degradation as a hard limit, and cannot explain why a diversely calibrated IQ3 preserves far more."}]},"prev":{"slug":"inside-a-gguf-file","title":"Inside a GGUF file","url":"https://addaly.com/learn/running-models-yourself/inside-a-gguf-file"},"next":{"slug":"gpu-quantisation-formats","title":"GPTQ, AWQ and the GPU-side formats","url":"https://addaly.com/learn/running-models-yourself/gpu-quantisation-formats"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'K-quants and importance matrices', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/k-quants-and-importance-matrices (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"K-quants and importance matrices","courseTitle":"Running Models Yourself","module":{"index":3,"slug":"quantisation-and-formats","title":"Quantisation and model formats"},"answer":null,"keyIdea":"K-quants spend bits where the model is sensitive and importance matrices decide where that is, so the calibration text silently determines which languages and tasks a low-bit quant preserves.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"rounding-is-not-the-hard-part","runs":[{"text":"Rounding is not the hard part"}]},{"t":"p","runs":[{"text":"Quantising a weight is trivial: divide by a scale, round, store the integer. The hard part is choosing the scales, and choosing which weights deserve protection. Everything that separates a good 4-bit model from a ruined one is in those two decisions."}]},{"t":"h2","id":"blocks-super-blocks-and-the-real-bit-rate","runs":[{"text":"Blocks, super-blocks and the real bit rate"}]},{"t":"p","runs":[{"text":"A simple scheme takes 32 weights, finds the largest magnitude among them, sets a scale so the range fits, and stores 32 four-bit integers plus one 16-bit scale. That is 4 + 16/32 = 4.5 bits per weight, and the extra half bit is the reason a “4-bit” file is never exactly half the size of an 8-bit one."}]},{"t":"p","runs":[{"text":"K-quants add a level. Groups of blocks form a super-block with its own scale, and the per-block scales are themselves quantised to 6 bits against the super-block scale. The result stores finer-grained scales for less overhead, so a block whose weights happen to be tiny is not forced to share a scale with a block containing an outlier. Outliers are the whole problem: one large weight in a block stretches the scale and coarsens every other weight in it."}]},{"t":"p","runs":[{"text":"This is also why block size is a trade you can see. Smaller blocks track the local distribution better and cost more scale storage. Group size 128 in GPTQ and AWQ is the same choice made differently."}]},{"t":"h2","id":"what-an-importance-matrix-does","runs":[{"text":"What an importance matrix does"}]},{"t":"p","runs":[{"text":"An importance matrix, "},{"text":"imatrix","code":true},{"text":" in llama.cpp, is computed by running the full-precision model over a body of text and recording, for each weight, how much the activations flowing through it contribute. Weights that are consistently multiplied by large activations matter more; an error there propagates. Weights that are rarely activated can be rounded harder."}]},{"t":"p","runs":[{"text":"The quantiser then uses that matrix to choose scales that minimise error weighted by importance, rather than raw error. The effect is small at 5 and 6 bits, clear at 4, and decisive below 4: the IQ2 and IQ3 schemes are essentially unusable without one."}]},{"t":"p","runs":[{"text":"Producing one:"}]},{"t":"code","lang":"bash","label":"Bash","text":"llama-imatrix -m model-f16.gguf -f calibration.txt -o model.imatrix --chunks 200\nllama-quantize --imatrix model.imatrix model-f16.gguf model-IQ3_M.gguf IQ3_M"},{"t":"p","runs":[{"text":"It takes minutes to an hour depending on the model and the amount of text."}]},{"t":"h2","id":"calibration-text-is-a-choice-with-consequences","runs":[{"text":"Calibration text is a choice with consequences"}]},{"t":"p","runs":[{"text":"This is the part most people never think about and it is where a real trap lives."}]},{"t":"p","runs":[{"text":"The importance matrix reflects the text you fed it. Calibrate on English Wikipedia and the weights that matter for English prose are protected. The weights that matter for Bengali, for Python, for JSON escaping or for long-form instruction following were not exercised, so the quantiser was free to damage them, and it did, because that is what it was told to do."}]},{"t":"p","runs":[{"text":"The practical consequences:"}]},{"t":"ul","items":[[{"text":"A general quant should be calibrated on diverse text","b":true},{"text":" (several languages, code, dialogue, structured output) and the well-known community quantisers do this deliberately."}],[{"text":"A quant for a specific job can be calibrated on that job’s text and be better than a general one at it","b":true},{"text":", while being worse at everything else. If you run one narrow task at very low bit rates, this is a genuine and underused optimisation."}],[{"text":"Never calibrate on your evaluation set.","b":true},{"text":" You will measure a model tuned to the exact text you are testing on, and the number will be meaningless."}]]},{"t":"p","runs":[{"text":"An honest limitation: the amount of calibration text used by most published quants is small (a few hundred thousand tokens is typical) which is enough to identify the broad importance structure and not enough to represent everything the model can do. This is one reason low-bit quants degrade unevenly across languages and tasks in ways that a single perplexity number does not reveal."}]},{"t":"h2","id":"choosing-a-tier-practically","runs":[{"text":"Choosing a tier, practically"}]},{"t":"ul","items":[[{"text":"Q8_0","b":true},{"text":": no imatrix needed, effectively lossless, half the size of fp16."}],[{"text":"Q6_K, Q5_K_M","b":true},{"text":": safe for essentially everything. Imatrix optional."}],[{"text":"Q4_K_M","b":true},{"text":": the standard trade. Imatrix helps slightly; take it if offered."}],[{"text":"IQ4_XS","b":true},{"text":": about the same quality as Q4_K_M at a slightly smaller size, needs an imatrix, more compute to decode."}],[{"text":"Q3_K_M and IQ3","b":true},{"text":": visible damage to precision-critical work. Imatrix now mandatory."}],[{"text":"IQ2","b":true},{"text":": only for very large models where the alternative is not running them at all."}]]},{"t":"p","runs":[{"text":"And the check that settles arguments: run your thirty-item audition set against two candidate quants. The difference between IQ4_XS and Q4_K_M is smaller than the difference between a good prompt and a bad one, and people spend far more time on the former."}]}],"check":{"question":[{"text":"A team quantises a multilingual model to IQ3_M using an importance matrix calibrated on English technical documentation. English output is fine; Hindi output degrades sharply compared with the Q5_K_M version. What is the mechanism?"}],"options":[{"text":[{"text":"Devanagari tokens occupy more of the vocabulary, so the output layer is under greater pressure at low bit rates regardless of calibration."}],"correct":false,"tempting":[{"text":"Identifies a genuine tokenisation cost that affects speed and context use, not a mechanism by which calibration language changes weight protection."}]},{"text":[{"text":"IQ schemes are known to be unsuitable for non-Latin scripts because their codebooks are derived from Latin-script statistics."}],"correct":false,"tempting":[{"text":"Invents a property of the format; the lattice codebooks are script-agnostic and it is the imatrix, not the scheme, that carries language bias."}]},{"text":[{"text":"The importance matrix recorded which weights matter for English text, so the weights carrying Hindi capability were treated as unimportant and rounded hardest."}],"correct":true,"tempting":null},{"text":[{"text":"Three-bit quantisation cannot represent multilingual embeddings at all, so any IQ3 build will lose non-English languages."}],"correct":false,"tempting":[{"text":"Overstates a gradual degradation as a hard limit, and cannot explain why a diversely calibrated IQ3 preserves far more."}]}],"order":[1,2,0,3],"why":null},"prev":{"slug":"inside-a-gguf-file","title":"Inside a GGUF file","minutes":8},"next":{"slug":"gpu-quantisation-formats","title":"GPTQ, AWQ and the GPU-side formats","minutes":9}}}