QLoRA: quantising the part you are not training

The idea in one line

The frozen base model receives no gradients, so it does not need to be stored precisely. Quantise it to 4 bits, keep the small trainable adapter in higher precision, and a 70B model becomes trainable on a single 48 GB card.

The numbers for an 8B model: bf16 weights are 16 GB, NF4 weights are about 4.1 GB. For a 70B: 140 GB becomes about 36 GB.

NF4, and why not plain int4

Uniform 4-bit quantisation carves the range into 16 evenly spaced buckets. Neural network weights are not uniformly distributed (they are roughly zero-centred and bell-shaped) so even spacing wastes most of its buckets on values that barely occur and crowds the region where the weights actually live.

NormalFloat 4 places its 16 levels so that, for a normally distributed tensor, each bucket receives an equal share of the values. It is information-theoretically matched to the distribution the weights actually have, and it measurably beats uniform int4 at the same bit width.

Two further pieces make it work:

Block-wise scaling. Quantisation is applied per block of 64 weights, each with its own scaling constant, rather than per tensor. One outlier weight then distorts 64 values instead of 16 million.

Double quantisation. Those per-block constants are themselves numbers you have to store: one fp32 per 64 weights is 0.5 bits per parameter of overhead. Quantising them to 8 bits, in blocks of 256, brings the overhead down to about 0.127 bits per parameter. On a 65B model that saving is around 3 GB, which is the difference between fitting and not.

What it costs

Three honest costs, none of them usually decisive but all of them real:

1. Speed. Every matrix multiplication dequantises its weights to bf16 on the fly. QLoRA typically runs 25 to 40% slower per step than the same LoRA on unquantised weights. You are trading time for the ability to run at all.
2. Quantisation error in the forward pass. The gradients computed for your adapter are gradients through a slightly wrong model. The QLoRA paper’s central claim is that the final quality matches 16-bit LoRA on the tasks measured, and that has held up broadly, but “broadly” is the honest word. On tasks that are sensitive to small numeric differences, and at very small model sizes where quantisation error is proportionally larger, the gap is not always zero.
3. Merging is awkward. Your adapter was fitted against the quantised base. Merge it into the full-precision base and you get a model that is not quite what you trained; merge it into the quantised base and you must requantise, which introduces its own error. The practical answers are to keep the adapter separate at serving time, or to use a method designed for this (QA-LoRA and similar) rather than merging naively and hoping.

The configuration

from transformers import BitsAndBytesConfig
import torch

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",          # not "fp4"
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,  # float16 on a T4
)

Four lines, and three of them have a wrong value that works. fp4 instead of nf4 trains without complaint and is slightly worse. Leaving double quantisation off costs memory you may need. Setting the compute dtype to fp32 quietly removes most of the speed benefit.

Pair it with optim="paged_adamw_8bit". Paged optimisers use CUDA unified memory so that a memory spike on one unusually long batch is absorbed by CPU RAM instead of ending the run.

When not to use it

If the model fits in bf16 on the card you have, use plain LoRA. QLoRA is a memory technique, and paying 30% throughput for memory you did not need is a straight loss. A 3B model on a 24 GB card has no business being quantised for training.

The decision is arithmetic, not taste: parameters × 2 bytes, plus activations, plus logits. If that is under your card’s capacity with headroom, skip the quantisation.

The free path

This is the technique that put fine-tuning on free hardware. A 7B or 8B model in NF4 at sequence length 1,024, batch 1 with gradient accumulation and checkpointing on, fits inside a free Colab or Kaggle T4’s 16 GB. It is slow (expect several hours for a few thousand steps) and Colab will disconnect you, so save checkpoints to Drive every couple of hundred steps. But it costs nothing, and the resulting adapter is the same kind of artefact a rented A100 would have produced.