Quantising the model you ship

Training quantisation and serving quantisation are different

QLoRA’s NF4 exists to make a frozen base fit in memory during training. It is not the format you serve in: it is comparatively slow at inference, because it was optimised for memory rather than throughput.

For serving, pick by where you are running:

GGUF, for CPU, laptops, phones and mixed CPU-GPU. The llama.cpp format, and what Ollama and LM Studio consume. Its k-quant variants give a fine-grained quality-size ladder. Q4_K_M is the widely used default and the usual sweet spot; Q5_K_M if you have the memory; Q8_0 when you want near-lossless.

AWQ, for GPU serving. Activation-aware weight quantisation protects the small fraction of weights that matter most to activations, and it is well supported by vLLM with good throughput at 4 bits.

GPTQ, the older calibration-based method. Still widely available, generally superseded by AWQ for new work.

FP8, on H100-class hardware. Close to lossless and fast, when you have the cards for it.

The sizes

For an 8B model, approximately:

fp16 / bf16        16.0 GB
Q8_0                8.5 GB
Q5_K_M              5.7 GB
Q4_K_M              4.9 GB
Q3_K_M              4.0 GB

The gap between fp16 and Q4_K_M is the difference between needing a data-centre card and running on a laptop.

What it costs in quality

At 4 bits on a model of 7B or more, the degradation is small: measurable on perplexity, often not visible on a task. Below 4 bits it becomes real, and Q3 and Q2 are noticeably worse in ways users notice.

Two effects deserve naming, because they are not general knowledge:

Smaller models suffer more. A 1B model at 4 bits loses proportionally more than an 8B model does. Quantisation error is roughly constant per weight, and a small model has less redundancy to absorb it. For anything under 3B, prefer 8-bit.

Quantisation can erase a fine-tune. This is the one that catches people. Your adapter changed the weights by a small amount: that is what a low-rank update of modest magnitude does. If the quantisation step’s error is of comparable size to your adapter’s delta, the changes are rounded away, and the quantised tuned model behaves like the quantised base model.

The symptom is a model that evaluated well before conversion and behaves like it was never trained afterwards. The diagnosis is a three-way comparison (base, tuned, tuned-and-quantised) on the same set, which takes minutes and is otherwise very confusing to debug.

The order of operations

# 1. merge the adapter into the fp16 base
python -c "from peft import AutoPeftModelForCausalLM; \
  AutoPeftModelForCausalLM.from_pretrained('out/adapter', torch_dtype='float16') \
  .merge_and_unload().save_pretrained('out/merged')"

# 2. convert, then quantise
python llama.cpp/convert_hf_to_gguf.py out/merged --outfile m-f16.gguf
./llama-quantize m-f16.gguf m-q4km.gguf Q4_K_M

Merge into the full-precision base, then quantise the merged model. Merging into an already-quantised base, or quantising and then merging, both introduce error the adapter was never fitted against.

If the adapter was trained with QLoRA, there is a subtlety worth stating: it was fitted against a 4-bit base, and merging it into the fp16 base is not exactly what it was trained for. In practice this usually works and occasionally loses a little quality. Measure it; that is what the three-way comparison is for.

Calibration data is not optional

AWQ and GPTQ need a calibration set to decide which weights matter. The default examples use general web text.

Use your own data instead: a few hundred examples from your task. The method is choosing what to protect based on which weights produce large activations on the calibration inputs, so calibrating on generic text protects the weights that matter for generic text. Calibrating on your data protects the weights that matter for your task, and the difference is measurable on task metrics.

The rule that covers all of it

Evaluate the exact artefact you will serve. Not the adapter, not the merged fp16 model, not the same quantisation of a different checkpoint. The file that will answer requests.

Quantisation, merging, format conversion and engine choice each change the model’s output. Any of them can lose you a point, and one of them can lose you the entire fine-tune. Your evaluation script should take a path and a backend and run against whatever is there, so that running it on the final file costs nothing and happens every time.