{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"quantising-a-model-yourself","title":"Quantising a model yourself","url":"https://addaly.com/learn/running-models-yourself/quantising-a-model-yourself","position":24,"of":82,"minutes":10,"answer":"Quantise from the original weights, never from another quant, and check the chat template and tokeniser survived the conversion: those two mistakes account for most locally produced models that appear to be broken.","keyIdea":"Quantise from the original weights, never from another quant, and check the chat template and tokeniser survived the conversion: those two mistakes account for most locally produced models that appear to be broken."},"body":{"format":"markdown","text":"## Why you would\n\nMost of the time somebody has already published the quant you want. Four situations where they have not:\n\n- You fine-tuned a model and need to serve it.\n- The published quants stop at Q4_K_M and your card would hold Q5.\n- You want an importance matrix calibrated on your own domain.\n- The available quants came from a model revision that was later fixed.\n\nAll four are an hour of work on a machine you already have.\n\n## GGUF, end to end\n\nStart from the original repository: the `.safetensors` files, not somebody else's GGUF. Quantising a quant compounds the loss.\n\n```bash\n# 1. get the weights\nhf download Qwen/Qwen3-8B --local-dir ./Qwen3-8B\n\n# 2. convert to a full-precision GGUF\npython3 llama.cpp/convert_hf_to_gguf.py ./Qwen3-8B \\\n  --outfile Qwen3-8B-f16.gguf --outtype f16\n\n# 3. quantise\n./llama.cpp/build/bin/llama-quantize \\\n  Qwen3-8B-f16.gguf Qwen3-8B-Q5_K_M.gguf Q5_K_M\n```\n\nStep 2 needs enough RAM to hold the model in fp16: about twice the parameter count in gigabytes. Step 3 is fast and needs much less. `llama-quantize` with no arguments lists every available type, which is the honest reference rather than a table that ages.\n\nWith an importance matrix, insert one step:\n\n```bash\n./llama.cpp/build/bin/llama-imatrix \\\n  -m Qwen3-8B-f16.gguf -f calibration.txt -o q3.imatrix --chunks 200\n\n./llama.cpp/build/bin/llama-quantize --imatrix q3.imatrix \\\n  Qwen3-8B-f16.gguf Qwen3-8B-IQ3_M.gguf IQ3_M\n```\n\nFor `calibration.txt`, a few hundred thousand tokens of text resembling your real use: your own documents, a slice of a public corpus, code if you generate code, several languages if you use several. Not your evaluation set.\n\n## AWQ\n\n```python\nfrom awq import AutoAWQForCausalLM\nfrom transformers import AutoTokenizer\n\nmodel = AutoAWQForCausalLM.from_pretrained(\"./Qwen3-8B\")\ntok = AutoTokenizer.from_pretrained(\"./Qwen3-8B\")\n\nmodel.quantize(tok, quant_config={\n    \"zero_point\": True, \"q_group_size\": 128,\n    \"w_bit\": 4, \"version\": \"GEMM\"\n})\nmodel.save_quantized(\"./Qwen3-8B-AWQ\")\ntok.save_pretrained(\"./Qwen3-8B-AWQ\")\n```\n\nThis needs a GPU and takes ten minutes to a couple of hours depending on size. The default calibration set is a slice of a public corpus; pass your own if your domain is unusual.\n\n## compressed-tensors with llm-compressor\n\nThe route for anything you intend to serve under vLLM, and the one that supports FP8, INT8 with activation quantisation, and 4-bit schemes from one tool:\n\n```python\nfrom llmcompressor.transformers import oneshot\nfrom llmcompressor.modifiers.quantization import QuantizationModifier\n\noneshot(\n    model=\"./Qwen3-8B\",\n    recipe=QuantizationModifier(targets=\"Linear\", scheme=\"FP8_DYNAMIC\",\n                                ignore=[\"lm_head\"]),\n    output_dir=\"./Qwen3-8B-FP8\",\n)\n```\n\nNote `ignore=[\"lm_head\"]`. The output layer is left at full precision in nearly every recipe, for the reason given earlier: its errors land directly on token probabilities.\n\n## Time, disk and what it actually runs on\n\nPlan the space first, because running out halfway through a conversion wastes the whole run. You need the original weights on disk at roughly two bytes per parameter (about 15 GB for a 7B model, 140 GB for a 70B) plus room for the intermediate 16-bit GGUF and then the quantised output. Three copies briefly coexist. A 70B conversion on a 256 GB drive is uncomfortably tight.\n\nThe time and hardware differ by format, which surprises people. GGUF conversion and quantisation are CPU and disk work: no GPU is needed at all, and a 7B model takes a few minutes on an ordinary machine. An importance-matrix pass is a forward pass over calibration text, so it does want a GPU and takes from several minutes to an hour depending on how much text you feed it. AWQ and GPTQ both require a GPU, because both search for scaling factors by running the model, and a 7B model typically takes twenty minutes to an hour on a consumer card.\n\nThat asymmetry is useful: anyone with no GPU can still produce GGUF quants of models they can barely run.\n\n## The mistakes that cost an afternoon\n\n**Quantising an already-quantised model.** Twice the damage, and the second pass optimises against the first pass's errors. Always start from the original.\n\n**Running out of RAM during conversion.** Conversion holds the fp16 model. An 8B needs about 16 GB; a 70B needs 140 GB, which usually means a rented machine for an hour or a sharded conversion.\n\n**Forgetting the tokeniser.** AWQ and compressed-tensors outputs need the tokeniser files saved alongside, or the model loads and produces nonsense.\n\n**A missing or wrong chat template.** After conversion, dump the metadata and check the template is present and correct. This is the most common defect in community quants and it is easy to reproduce yourself.\n\n**Not testing before publishing.** Run your audition set against the new file. Fifteen minutes, and it catches every one of the above.\n\n## If you publish it\n\nSay which base revision you started from, which tool and version you used, whether an importance matrix was used and what it was calibrated on, and any evaluation you ran. That is four lines in a README and it is the difference between a quant somebody can trust and one they have to test from scratch. The licence of the original model comes with the file, and if the original required attribution or a naming convention, so does yours."},"check":{"question":"Someone downloads a Q8_0 GGUF, converts it back to fp16, and quantises the result to Q4_K_M to save disk space. The output is noticeably worse than a Q4_K_M made by others from the original weights. Why?","options":[{"text":"The intermediate fp16 conversion loses the k-quant scale hierarchy, which cannot be reconstructed from dequantised weights.","correct":false,"tempting":"Describes something that is true but harmless (the second pass computes its own scales) and misses that the damage is already in the values."},{"text":"Converting Q8 to fp16 is not supported by the tooling, so the intermediate file contains garbage in the promoted tensors.","correct":false,"tempting":"Asserts a tooling failure that would produce obvious breakage rather than a modest quality gap."},{"text":"The Q8 rounding errors are baked into the weights, so the second quantiser is fitting scales to already-damaged values and the two losses compound.","correct":true},{"text":"Q8_0 uses a different block size from Q4_K_M, so the block boundaries no longer align and each 4-bit block spans two Q8 scales.","correct":false,"tempting":"Invents an alignment problem; the second pass reads plain floating-point values with no memory of the earlier blocking."}]},"prev":{"slug":"fp8-and-native-low-precision","title":"FP8 and models born quantised","url":"https://addaly.com/learn/running-models-yourself/fp8-and-native-low-precision"},"next":{"slug":"measuring-quantisation-damage","title":"Measuring the damage, properly","url":"https://addaly.com/learn/running-models-yourself/measuring-quantisation-damage"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Quantising a model yourself', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/quantising-a-model-yourself (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Quantising a model yourself","courseTitle":"Running Models Yourself","module":{"index":3,"slug":"quantisation-and-formats","title":"Quantisation and model formats"},"answer":null,"keyIdea":"Quantise from the original weights, never from another quant, and check the chat template and tokeniser survived the conversion: those two mistakes account for most locally produced models that appear to be broken.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-you-would","runs":[{"text":"Why you would"}]},{"t":"p","runs":[{"text":"Most of the time somebody has already published the quant you want. Four situations where they have not:"}]},{"t":"ul","items":[[{"text":"You fine-tuned a model and need to serve it."}],[{"text":"The published quants stop at Q4_K_M and your card would hold Q5."}],[{"text":"You want an importance matrix calibrated on your own domain."}],[{"text":"The available quants came from a model revision that was later fixed."}]]},{"t":"p","runs":[{"text":"All four are an hour of work on a machine you already have."}]},{"t":"h2","id":"gguf-end-to-end","runs":[{"text":"GGUF, end to end"}]},{"t":"p","runs":[{"text":"Start from the original repository: the "},{"text":".safetensors","code":true},{"text":" files, not somebody else’s GGUF. Quantising a quant compounds the loss."}]},{"t":"code","lang":"bash","label":"Bash","text":"# 1. get the weights\nhf download Qwen/Qwen3-8B --local-dir ./Qwen3-8B\n\n# 2. convert to a full-precision GGUF\npython3 llama.cpp/convert_hf_to_gguf.py ./Qwen3-8B \\\n  --outfile Qwen3-8B-f16.gguf --outtype f16\n\n# 3. quantise\n./llama.cpp/build/bin/llama-quantize \\\n  Qwen3-8B-f16.gguf Qwen3-8B-Q5_K_M.gguf Q5_K_M"},{"t":"p","runs":[{"text":"Step 2 needs enough RAM to hold the model in fp16: about twice the parameter count in gigabytes. Step 3 is fast and needs much less. "},{"text":"llama-quantize","code":true},{"text":" with no arguments lists every available type, which is the honest reference rather than a table that ages."}]},{"t":"p","runs":[{"text":"With an importance matrix, insert one step:"}]},{"t":"code","lang":"bash","label":"Bash","text":"./llama.cpp/build/bin/llama-imatrix \\\n  -m Qwen3-8B-f16.gguf -f calibration.txt -o q3.imatrix --chunks 200\n\n./llama.cpp/build/bin/llama-quantize --imatrix q3.imatrix \\\n  Qwen3-8B-f16.gguf Qwen3-8B-IQ3_M.gguf IQ3_M"},{"t":"p","runs":[{"text":"For "},{"text":"calibration.txt","code":true},{"text":", a few hundred thousand tokens of text resembling your real use: your own documents, a slice of a public corpus, code if you generate code, several languages if you use several. Not your evaluation set."}]},{"t":"h2","id":"awq","runs":[{"text":"AWQ"}]},{"t":"code","lang":"python","label":"Python","text":"from awq import AutoAWQForCausalLM\nfrom transformers import AutoTokenizer\n\nmodel = AutoAWQForCausalLM.from_pretrained(\"./Qwen3-8B\")\ntok = AutoTokenizer.from_pretrained(\"./Qwen3-8B\")\n\nmodel.quantize(tok, quant_config={\n    \"zero_point\": True, \"q_group_size\": 128,\n    \"w_bit\": 4, \"version\": \"GEMM\"\n})\nmodel.save_quantized(\"./Qwen3-8B-AWQ\")\ntok.save_pretrained(\"./Qwen3-8B-AWQ\")"},{"t":"p","runs":[{"text":"This needs a GPU and takes ten minutes to a couple of hours depending on size. The default calibration set is a slice of a public corpus; pass your own if your domain is unusual."}]},{"t":"h2","id":"compressed-tensors-with-llm-compressor","runs":[{"text":"compressed-tensors with llm-compressor"}]},{"t":"p","runs":[{"text":"The route for anything you intend to serve under vLLM, and the one that supports FP8, INT8 with activation quantisation, and 4-bit schemes from one tool:"}]},{"t":"code","lang":"python","label":"Python","text":"from llmcompressor.transformers import oneshot\nfrom llmcompressor.modifiers.quantization import QuantizationModifier\n\noneshot(\n    model=\"./Qwen3-8B\",\n    recipe=QuantizationModifier(targets=\"Linear\", scheme=\"FP8_DYNAMIC\",\n                                ignore=[\"lm_head\"]),\n    output_dir=\"./Qwen3-8B-FP8\",\n)"},{"t":"p","runs":[{"text":"Note "},{"text":"ignore=[\"lm_head\"]","code":true},{"text":". The output layer is left at full precision in nearly every recipe, for the reason given earlier: its errors land directly on token probabilities."}]},{"t":"h2","id":"time-disk-and-what-it-actually-runs-on","runs":[{"text":"Time, disk and what it actually runs on"}]},{"t":"p","runs":[{"text":"Plan the space first, because running out halfway through a conversion wastes the whole run. You need the original weights on disk at roughly two bytes per parameter (about 15 GB for a 7B model, 140 GB for a 70B) plus room for the intermediate 16-bit GGUF and then the quantised output. Three copies briefly coexist. A 70B conversion on a 256 GB drive is uncomfortably tight."}]},{"t":"p","runs":[{"text":"The time and hardware differ by format, which surprises people. GGUF conversion and quantisation are CPU and disk work: no GPU is needed at all, and a 7B model takes a few minutes on an ordinary machine. An importance-matrix pass is a forward pass over calibration text, so it does want a GPU and takes from several minutes to an hour depending on how much text you feed it. AWQ and GPTQ both require a GPU, because both search for scaling factors by running the model, and a 7B model typically takes twenty minutes to an hour on a consumer card."}]},{"t":"p","runs":[{"text":"That asymmetry is useful: anyone with no GPU can still produce GGUF quants of models they can barely run."}]},{"t":"h2","id":"the-mistakes-that-cost-an-afternoon","runs":[{"text":"The mistakes that cost an afternoon"}]},{"t":"p","runs":[{"text":"Quantising an already-quantised model.","b":true},{"text":" Twice the damage, and the second pass optimises against the first pass’s errors. Always start from the original."}]},{"t":"p","runs":[{"text":"Running out of RAM during conversion.","b":true},{"text":" Conversion holds the fp16 model. An 8B needs about 16 GB; a 70B needs 140 GB, which usually means a rented machine for an hour or a sharded conversion."}]},{"t":"p","runs":[{"text":"Forgetting the tokeniser.","b":true},{"text":" AWQ and compressed-tensors outputs need the tokeniser files saved alongside, or the model loads and produces nonsense."}]},{"t":"p","runs":[{"text":"A missing or wrong chat template.","b":true},{"text":" After conversion, dump the metadata and check the template is present and correct. This is the most common defect in community quants and it is easy to reproduce yourself."}]},{"t":"p","runs":[{"text":"Not testing before publishing.","b":true},{"text":" Run your audition set against the new file. Fifteen minutes, and it catches every one of the above."}]},{"t":"h2","id":"if-you-publish-it","runs":[{"text":"If you publish it"}]},{"t":"p","runs":[{"text":"Say which base revision you started from, which tool and version you used, whether an importance matrix was used and what it was calibrated on, and any evaluation you ran. That is four lines in a README and it is the difference between a quant somebody can trust and one they have to test from scratch. The licence of the original model comes with the file, and if the original required attribution or a naming convention, so does yours."}]}],"check":{"question":[{"text":"Someone downloads a Q8_0 GGUF, converts it back to fp16, and quantises the result to Q4_K_M to save disk space. The output is noticeably worse than a Q4_K_M made by others from the original weights. Why?"}],"options":[{"text":[{"text":"The intermediate fp16 conversion loses the k-quant scale hierarchy, which cannot be reconstructed from dequantised weights."}],"correct":false,"tempting":[{"text":"Describes something that is true but harmless (the second pass computes its own scales) and misses that the damage is already in the values."}]},{"text":[{"text":"Converting Q8 to fp16 is not supported by the tooling, so the intermediate file contains garbage in the promoted tensors."}],"correct":false,"tempting":[{"text":"Asserts a tooling failure that would produce obvious breakage rather than a modest quality gap."}]},{"text":[{"text":"The Q8 rounding errors are baked into the weights, so the second quantiser is fitting scales to already-damaged values and the two losses compound."}],"correct":true,"tempting":null},{"text":[{"text":"Q8_0 uses a different block size from Q4_K_M, so the block boundaries no longer align and each 4-bit block spans two Q8 scales."}],"correct":false,"tempting":[{"text":"Invents an alignment problem; the second pass reads plain floating-point values with no memory of the earlier blocking."}]}],"order":[0,3,2,1],"why":null},"prev":{"slug":"fp8-and-native-low-precision","title":"FP8 and models born quantised","minutes":8},"next":{"slug":"measuring-quantisation-damage","title":"Measuring the damage, properly","minutes":9}}}