Quantization: what you buy and what you pay

The idea in one paragraph

Weights are stored as 16-bit numbers because training needed that precision. Inference mostly does not. Quantization stores each weight in fewer bits (8, 4, sometimes 3) by keeping a scale factor per small group of weights and storing each weight as a small integer offset from it. Memory falls roughly in proportion to the bit width. Accuracy falls too, by an amount that depends on the scheme, the bit width, and heavily on the task.

The four routes you will meet

bitsandbytes: quantizes on the fly as the model loads, so it works with any transformers model with no prepared repository:

from transformers import BitsAndBytesConfig

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(name, quantization_config=bnb,
                                             device_map="auto")

The easiest route and the standard one for QLoRA training. Its weakness is speed: 4-bit bitsandbytes inference is often slower than bfloat16 on a card with enough memory, because weights are dequantized on the fly. You are buying memory, not throughput.

AWQ and GPTQ: pre-quantized repositories, produced once by running a calibration set through the model to decide which weights matter. Loading is a normal from_pretrained. These are faster at inference than bitsandbytes because they use kernels written for the packed format, and they are the usual choice for GPU serving. The cost is that somebody had to produce the repository, and its quality depends on a calibration set you did not see.

GGUF: the llama.cpp format, covered in the next lesson. CPU-first, excellent on Apple silicon, the practical answer for a laptop.

Native low-precision formats: newer hardware supports 8-bit floating point directly, and models are increasingly published in it. Where the hardware supports it, this is quantization with much less quality loss, and it is the direction the ecosystem is moving.

What it actually costs in quality

Published perplexity comparisons and community measurements land in roughly the same place, with wide variation by model:

• 8-bit: the difference is usually below what you can detect on ordinary tasks. Treat it as close to free.
• 4-bit (nf4, AWQ, Q4_K_M): small degradation on chat and summarisation, larger on code, arithmetic, long structured output and instruction-following at the edges. Small models suffer more than large ones, because they have less redundancy to spare.
• 3-bit and below: visible damage. Sometimes still worth it if the alternative is not running at all.

The pattern worth holding: a 4-bit large model usually beats a 16-bit small model at the same memory budget. A 4-bit 13B model and an fp16 7B model both take about 8 GB, and the 13B is generally better. This is counter-intuitive and it is one of the most useful facts in the whole area.

The failure mode nobody warns about

Quantization damage is not uniform. It concentrates in exactly the capabilities that are hardest to spot-check: multi-step arithmetic, precise JSON, rarely used languages, and long-range consistency. A model that chats indistinguishably at 4-bit may have quietly got worse at emitting valid JSON, which your parser turns into an error rate three weeks later.

If your application depends on structure or arithmetic, measure it. Thirty examples with exact-match scoring at both precisions takes twenty minutes and is the difference between a decision and a guess.

Calibration, and what it means for you

AWQ and GPTQ choose their scales by watching activations on a calibration set: typically a few hundred samples of general web text. If your domain is far from that text, the quantization was tuned for a distribution you are not in. This is rarely catastrophic and it is a real reason a quantized model can underperform its published numbers on specialised text.

You can quantize with your own calibration data using the same libraries. For most people this is not worth it; for a narrow domain with a model you will run a million times, it is.

Choosing, in one pass

• Does it fit at bfloat16? Use bfloat16. Quantization is a constraint, not an improvement.
• GPU, needs to fit, needs speed? AWQ or GPTQ from a prepared repository.
• GPU, needs to fit, will be fine-tuned? bitsandbytes 4-bit plus LoRA.
• CPU, laptop or Mac? GGUF at Q4_K_M, or Q5_K_M if the memory is there.
• Structured output or maths matters? 8-bit, and measure.

Do this now

Take one model you can run at both bfloat16 and 4-bit. Run the same twenty prompts through each, including three that require exact output: a date reformatted, a small sum, a JSON object with four fields. Diff the results. The chat answers will look equivalent; the exact ones are where you will find the cost.