Quantising a model yourself

Why you would

Most of the time somebody has already published the quant you want. Four situations where they have not:

• You fine-tuned a model and need to serve it.
• The published quants stop at Q4_K_M and your card would hold Q5.
• You want an importance matrix calibrated on your own domain.
• The available quants came from a model revision that was later fixed.

All four are an hour of work on a machine you already have.

GGUF, end to end

Start from the original repository: the .safetensors files, not somebody else’s GGUF. Quantising a quant compounds the loss.

# 1. get the weights
hf download Qwen/Qwen3-8B --local-dir ./Qwen3-8B

# 2. convert to a full-precision GGUF
python3 llama.cpp/convert_hf_to_gguf.py ./Qwen3-8B \
  --outfile Qwen3-8B-f16.gguf --outtype f16

# 3. quantise
./llama.cpp/build/bin/llama-quantize \
  Qwen3-8B-f16.gguf Qwen3-8B-Q5_K_M.gguf Q5_K_M

Step 2 needs enough RAM to hold the model in fp16: about twice the parameter count in gigabytes. Step 3 is fast and needs much less. llama-quantize with no arguments lists every available type, which is the honest reference rather than a table that ages.

With an importance matrix, insert one step:

./llama.cpp/build/bin/llama-imatrix \
  -m Qwen3-8B-f16.gguf -f calibration.txt -o q3.imatrix --chunks 200

./llama.cpp/build/bin/llama-quantize --imatrix q3.imatrix \
  Qwen3-8B-f16.gguf Qwen3-8B-IQ3_M.gguf IQ3_M

For calibration.txt, a few hundred thousand tokens of text resembling your real use: your own documents, a slice of a public corpus, code if you generate code, several languages if you use several. Not your evaluation set.

AWQ

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model = AutoAWQForCausalLM.from_pretrained("./Qwen3-8B")
tok = AutoTokenizer.from_pretrained("./Qwen3-8B")

model.quantize(tok, quant_config={
    "zero_point": True, "q_group_size": 128,
    "w_bit": 4, "version": "GEMM"
})
model.save_quantized("./Qwen3-8B-AWQ")
tok.save_pretrained("./Qwen3-8B-AWQ")

This needs a GPU and takes ten minutes to a couple of hours depending on size. The default calibration set is a slice of a public corpus; pass your own if your domain is unusual.

compressed-tensors with llm-compressor

The route for anything you intend to serve under vLLM, and the one that supports FP8, INT8 with activation quantisation, and 4-bit schemes from one tool:

from llmcompressor.transformers import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

oneshot(
    model="./Qwen3-8B",
    recipe=QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC",
                                ignore=["lm_head"]),
    output_dir="./Qwen3-8B-FP8",
)

Note ignore=["lm_head"]. The output layer is left at full precision in nearly every recipe, for the reason given earlier: its errors land directly on token probabilities.

Time, disk and what it actually runs on

Plan the space first, because running out halfway through a conversion wastes the whole run. You need the original weights on disk at roughly two bytes per parameter (about 15 GB for a 7B model, 140 GB for a 70B) plus room for the intermediate 16-bit GGUF and then the quantised output. Three copies briefly coexist. A 70B conversion on a 256 GB drive is uncomfortably tight.

The time and hardware differ by format, which surprises people. GGUF conversion and quantisation are CPU and disk work: no GPU is needed at all, and a 7B model takes a few minutes on an ordinary machine. An importance-matrix pass is a forward pass over calibration text, so it does want a GPU and takes from several minutes to an hour depending on how much text you feed it. AWQ and GPTQ both require a GPU, because both search for scaling factors by running the model, and a 7B model typically takes twenty minutes to an hour on a consumer card.

That asymmetry is useful: anyone with no GPU can still produce GGUF quants of models they can barely run.

The mistakes that cost an afternoon

Quantising an already-quantised model. Twice the damage, and the second pass optimises against the first pass’s errors. Always start from the original.

Running out of RAM during conversion. Conversion holds the fp16 model. An 8B needs about 16 GB; a 70B needs 140 GB, which usually means a rented machine for an hour or a sharded conversion.

Forgetting the tokeniser. AWQ and compressed-tensors outputs need the tokeniser files saved alongside, or the model loads and produces nonsense.

A missing or wrong chat template. After conversion, dump the metadata and check the template is present and correct. This is the most common defect in community quants and it is easy to reproduce yourself.

Not testing before publishing. Run your audition set against the new file. Fifteen minutes, and it catches every one of the above.

If you publish it

Say which base revision you started from, which tool and version you used, whether an importance matrix was used and what it was calibrated on, and any evaluation you ran. That is four lines in a README and it is the difference between a quant somebody can trust and one they have to test from scratch. The licence of the original model comes with the file, and if the original required attribution or a naming convention, so does yours.