{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"quantising-for-serving","title":"Quantising the model you ship","url":"https://addaly.com/learn/fine-tuning/quantising-for-serving","position":77,"of":84,"minutes":9,"answer":"Serve with AWQ on a GPU or Q4_K_M in GGUF on a laptop, merge into the full-precision base before quantising, calibrate on your own data, and run a three-way base, tuned, tuned-and-quantised comparison.","keyIdea":"Serve with AWQ on a GPU or Q4_K_M in GGUF on a laptop, merge into the full-precision base before quantising, calibrate on your own data, and run a three-way base, tuned, tuned-and-quantised comparison, because quantisation error can be as large as your adapter's delta and round the entire fine-tune away."},"body":{"format":"markdown","text":"## Training quantisation and serving quantisation are different\n\nQLoRA's NF4 exists to make a frozen base fit in memory during training. It is not the format you serve in: it is comparatively slow at inference, because it was optimised for memory rather than throughput.\n\nFor serving, pick by where you are running:\n\n**GGUF**, for CPU, laptops, phones and mixed CPU-GPU. The `llama.cpp` format, and what Ollama and LM Studio consume. Its k-quant variants give a fine-grained quality-size ladder. `Q4_K_M` is the widely used default and the usual sweet spot; `Q5_K_M` if you have the memory; `Q8_0` when you want near-lossless.\n\n**AWQ**, for GPU serving. Activation-aware weight quantisation protects the small fraction of weights that matter most to activations, and it is well supported by vLLM with good throughput at 4 bits.\n\n**GPTQ**, the older calibration-based method. Still widely available, generally superseded by AWQ for new work.\n\n**FP8**, on H100-class hardware. Close to lossless and fast, when you have the cards for it.\n\n## The sizes\n\nFor an 8B model, approximately:\n\n```\nfp16 / bf16        16.0 GB\nQ8_0                8.5 GB\nQ5_K_M              5.7 GB\nQ4_K_M              4.9 GB\nQ3_K_M              4.0 GB\n```\n\nThe gap between fp16 and Q4_K_M is the difference between needing a data-centre card and running on a laptop.\n\n## What it costs in quality\n\nAt 4 bits on a model of 7B or more, the degradation is small: measurable on perplexity, often not visible on a task. Below 4 bits it becomes real, and Q3 and Q2 are noticeably worse in ways users notice.\n\nTwo effects deserve naming, because they are not general knowledge:\n\n**Smaller models suffer more.** A 1B model at 4 bits loses proportionally more than an 8B model does. Quantisation error is roughly constant per weight, and a small model has less redundancy to absorb it. For anything under 3B, prefer 8-bit.\n\n**Quantisation can erase a fine-tune.** This is the one that catches people. Your adapter changed the weights by a small amount: that is what a low-rank update of modest magnitude does. If the quantisation step's error is of comparable size to your adapter's delta, the changes are rounded away, and the quantised tuned model behaves like the quantised base model.\n\nThe symptom is a model that evaluated well before conversion and behaves like it was never trained afterwards. The diagnosis is a three-way comparison (base, tuned, tuned-and-quantised) on the same set, which takes minutes and is otherwise very confusing to debug.\n\n## The order of operations\n\n```bash\n# 1. merge the adapter into the fp16 base\npython -c \"from peft import AutoPeftModelForCausalLM; \\\n  AutoPeftModelForCausalLM.from_pretrained('out/adapter', torch_dtype='float16') \\\n  .merge_and_unload().save_pretrained('out/merged')\"\n\n# 2. convert, then quantise\npython llama.cpp/convert_hf_to_gguf.py out/merged --outfile m-f16.gguf\n./llama-quantize m-f16.gguf m-q4km.gguf Q4_K_M\n```\n\nMerge into the **full-precision** base, then quantise the merged model. Merging into an already-quantised base, or quantising and then merging, both introduce error the adapter was never fitted against.\n\nIf the adapter was trained with QLoRA, there is a subtlety worth stating: it was fitted against a 4-bit base, and merging it into the fp16 base is not exactly what it was trained for. In practice this usually works and occasionally loses a little quality. Measure it; that is what the three-way comparison is for.\n\n## Calibration data is not optional\n\nAWQ and GPTQ need a calibration set to decide which weights matter. The default examples use general web text.\n\nUse **your own data** instead: a few hundred examples from your task. The method is choosing what to protect based on which weights produce large activations on the calibration inputs, so calibrating on generic text protects the weights that matter for generic text. Calibrating on your data protects the weights that matter for your task, and the difference is measurable on task metrics.\n\n## The rule that covers all of it\n\nEvaluate the exact artefact you will serve. Not the adapter, not the merged fp16 model, not the same quantisation of a different checkpoint. The file that will answer requests.\n\nQuantisation, merging, format conversion and engine choice each change the model's output. Any of them can lose you a point, and one of them can lose you the entire fine-tune. Your evaluation script should take a path and a backend and run against whatever is there, so that running it on the final file costs nothing and happens every time."},"check":{"question":"A tuned 1.5B model evaluates at 0.88 in bf16 and at 0.71 after quantising to 4 bits, while the base model scores 0.70 both before and after. What is the likeliest mechanism?","options":[{"text":"The adapter's weight changes are small relative to quantisation error at this size, so they rounded away.","correct":true},{"text":"The calibration set was too small, so the quantiser chose poor scaling factors for the whole model.","correct":false,"tempting":"Bad calibration would degrade the base model too, which held steady at 0.70 across both precisions."},{"text":"Quantisation removed the adapter, since merged weights revert to base values when the model is converted.","correct":false,"tempting":"Imagines a mechanism that reverts weights; merging writes the deltas into the matrix and conversion has no notion of what came from an adapter."},{"text":"Four-bit quantisation always costs about 17 points of accuracy, which is why 8-bit is the standard for production.","correct":false,"tempting":"Asserts a fixed penalty that does not match the base model's unchanged score in the same experiment."}]},"prev":{"slug":"serving-the-adapter","title":"Serving the Adapter","url":"https://addaly.com/learn/fine-tuning/serving-the-adapter"},"next":{"slug":"many-adapters-at-once","title":"Serving many adapters from one base","url":"https://addaly.com/learn/fine-tuning/many-adapters-at-once"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Quantising the model you ship', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/quantising-for-serving (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Quantising the model you ship","courseTitle":"Fine-Tuning, and When Not To","module":{"index":9,"slug":"shipping-and-living-with-it","title":"Shipping it, and living with it"},"answer":null,"keyIdea":"Serve with AWQ on a GPU or Q4_K_M in GGUF on a laptop, merge into the full-precision base before quantising, calibrate on your own data, and run a three-way base, tuned, tuned-and-quantised comparison, because quantisation error can be as large as your adapter’s delta and round the entire fine-tune away.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"training-quantisation-and-serving-quantisation-are-different","runs":[{"text":"Training quantisation and serving quantisation are different"}]},{"t":"p","runs":[{"text":"QLoRA’s NF4 exists to make a frozen base fit in memory during training. It is not the format you serve in: it is comparatively slow at inference, because it was optimised for memory rather than throughput."}]},{"t":"p","runs":[{"text":"For serving, pick by where you are running:"}]},{"t":"p","runs":[{"text":"GGUF","b":true},{"text":", for CPU, laptops, phones and mixed CPU-GPU. The "},{"text":"llama.cpp","code":true},{"text":" format, and what Ollama and LM Studio consume. Its k-quant variants give a fine-grained quality-size ladder. "},{"text":"Q4_K_M","code":true},{"text":" is the widely used default and the usual sweet spot; "},{"text":"Q5_K_M","code":true},{"text":" if you have the memory; "},{"text":"Q8_0","code":true},{"text":" when you want near-lossless."}]},{"t":"p","runs":[{"text":"AWQ","b":true},{"text":", for GPU serving. Activation-aware weight quantisation protects the small fraction of weights that matter most to activations, and it is well supported by vLLM with good throughput at 4 bits."}]},{"t":"p","runs":[{"text":"GPTQ","b":true},{"text":", the older calibration-based method. Still widely available, generally superseded by AWQ for new work."}]},{"t":"p","runs":[{"text":"FP8","b":true},{"text":", on H100-class hardware. Close to lossless and fast, when you have the cards for it."}]},{"t":"h2","id":"the-sizes","runs":[{"text":"The sizes"}]},{"t":"p","runs":[{"text":"For an 8B model, approximately:"}]},{"t":"code","lang":"","label":"Text","text":"fp16 / bf16        16.0 GB\nQ8_0                8.5 GB\nQ5_K_M              5.7 GB\nQ4_K_M              4.9 GB\nQ3_K_M              4.0 GB"},{"t":"p","runs":[{"text":"The gap between fp16 and Q4_K_M is the difference between needing a data-centre card and running on a laptop."}]},{"t":"h2","id":"what-it-costs-in-quality","runs":[{"text":"What it costs in quality"}]},{"t":"p","runs":[{"text":"At 4 bits on a model of 7B or more, the degradation is small: measurable on perplexity, often not visible on a task. Below 4 bits it becomes real, and Q3 and Q2 are noticeably worse in ways users notice."}]},{"t":"p","runs":[{"text":"Two effects deserve naming, because they are not general knowledge:"}]},{"t":"p","runs":[{"text":"Smaller models suffer more.","b":true},{"text":" A 1B model at 4 bits loses proportionally more than an 8B model does. Quantisation error is roughly constant per weight, and a small model has less redundancy to absorb it. For anything under 3B, prefer 8-bit."}]},{"t":"p","runs":[{"text":"Quantisation can erase a fine-tune.","b":true},{"text":" This is the one that catches people. Your adapter changed the weights by a small amount: that is what a low-rank update of modest magnitude does. If the quantisation step’s error is of comparable size to your adapter’s delta, the changes are rounded away, and the quantised tuned model behaves like the quantised base model."}]},{"t":"p","runs":[{"text":"The symptom is a model that evaluated well before conversion and behaves like it was never trained afterwards. The diagnosis is a three-way comparison (base, tuned, tuned-and-quantised) on the same set, which takes minutes and is otherwise very confusing to debug."}]},{"t":"h2","id":"the-order-of-operations","runs":[{"text":"The order of operations"}]},{"t":"code","lang":"bash","label":"Bash","text":"# 1. merge the adapter into the fp16 base\npython -c \"from peft import AutoPeftModelForCausalLM; \\\n  AutoPeftModelForCausalLM.from_pretrained('out/adapter', torch_dtype='float16') \\\n  .merge_and_unload().save_pretrained('out/merged')\"\n\n# 2. convert, then quantise\npython llama.cpp/convert_hf_to_gguf.py out/merged --outfile m-f16.gguf\n./llama-quantize m-f16.gguf m-q4km.gguf Q4_K_M"},{"t":"p","runs":[{"text":"Merge into the "},{"text":"full-precision","b":true},{"text":" base, then quantise the merged model. Merging into an already-quantised base, or quantising and then merging, both introduce error the adapter was never fitted against."}]},{"t":"p","runs":[{"text":"If the adapter was trained with QLoRA, there is a subtlety worth stating: it was fitted against a 4-bit base, and merging it into the fp16 base is not exactly what it was trained for. In practice this usually works and occasionally loses a little quality. Measure it; that is what the three-way comparison is for."}]},{"t":"h2","id":"calibration-data-is-not-optional","runs":[{"text":"Calibration data is not optional"}]},{"t":"p","runs":[{"text":"AWQ and GPTQ need a calibration set to decide which weights matter. The default examples use general web text."}]},{"t":"p","runs":[{"text":"Use "},{"text":"your own data","b":true},{"text":" instead: a few hundred examples from your task. The method is choosing what to protect based on which weights produce large activations on the calibration inputs, so calibrating on generic text protects the weights that matter for generic text. Calibrating on your data protects the weights that matter for your task, and the difference is measurable on task metrics."}]},{"t":"h2","id":"the-rule-that-covers-all-of-it","runs":[{"text":"The rule that covers all of it"}]},{"t":"p","runs":[{"text":"Evaluate the exact artefact you will serve. Not the adapter, not the merged fp16 model, not the same quantisation of a different checkpoint. The file that will answer requests."}]},{"t":"p","runs":[{"text":"Quantisation, merging, format conversion and engine choice each change the model’s output. Any of them can lose you a point, and one of them can lose you the entire fine-tune. Your evaluation script should take a path and a backend and run against whatever is there, so that running it on the final file costs nothing and happens every time."}]}],"check":{"question":[{"text":"A tuned 1.5B model evaluates at 0.88 in bf16 and at 0.71 after quantising to 4 bits, while the base model scores 0.70 both before and after. What is the likeliest mechanism?"}],"options":[{"text":[{"text":"The adapter’s weight changes are small relative to quantisation error at this size, so they rounded away."}],"correct":true,"tempting":null},{"text":[{"text":"The calibration set was too small, so the quantiser chose poor scaling factors for the whole model."}],"correct":false,"tempting":[{"text":"Bad calibration would degrade the base model too, which held steady at 0.70 across both precisions."}]},{"text":[{"text":"Quantisation removed the adapter, since merged weights revert to base values when the model is converted."}],"correct":false,"tempting":[{"text":"Imagines a mechanism that reverts weights; merging writes the deltas into the matrix and conversion has no notion of what came from an adapter."}]},{"text":[{"text":"Four-bit quantisation always costs about 17 points of accuracy, which is why 8-bit is the standard for production."}],"correct":false,"tempting":[{"text":"Asserts a fixed penalty that does not match the base model’s unchanged score in the same experiment."}]}],"order":[2,3,0,1],"why":null},"prev":{"slug":"serving-the-adapter","title":"Serving the Adapter","minutes":9},"next":{"slug":"many-adapters-at-once","title":"Serving many adapters from one base","minutes":8}}}