Fitting a fine-tune in 16 GB

Where the memory goes in training

Inference needs weights plus a cache. Training needs considerably more, and knowing the four components tells you exactly which dial to turn when it will not fit.

For a full fine-tune of a 7B model in mixed precision:

• Weights, 2 bytes each: 14 GB
• Gradients, one per weight, 2 bytes: 14 GB
• Optimiser state, Adam keeps two moments in 32-bit plus a master copy: roughly 12 bytes per parameter, 84 GB
• Activations, everything saved for the backward pass: depends on batch size and sequence length, often tens of GB

That is 110 GB or more, which means several datacentre GPUs. It is why almost nobody full-tunes a 7B model.

What QLoRA removes

QLoRA combines three ideas and the result fits on a free notebook GPU.

The base is quantised to 4-bit and frozen. Weights drop from 14 GB to about 3.5 GB. Because the base is never updated, quantisation error does not accumulate: it is a fixed distortion the adapter learns against.

Only the adapter trains. With 40 million trainable parameters instead of 7 billion, gradients and optimiser state fall from 98 GB to under a gigabyte.

Paged optimiser state moves optimiser memory to CPU RAM when a spike would otherwise overflow, which turns an occasional crash into an occasional slowdown.

The format is NF4, a 4-bit representation shaped for normally distributed values, plus double quantisation, which quantises the block scales themselves for a further small saving.

New total for a 7B: about 3.5 GB of base, under 1 GB of adapter and optimiser, plus activations. On a 16 GB card that leaves 11 GB for activations, which is the number you now control.

Activations are the dial you actually turn

Activation memory scales with batch size × sequence length, roughly linearly in both. When a run fails with an out-of-memory error, this is nearly always what to change.

Gradient checkpointing is the biggest single lever. Instead of storing every intermediate activation for the backward pass, it stores a few and recomputes the rest. Memory falls a lot; compute rises by roughly 30%. Turn it on by default.

Gradient accumulation gives you a large effective batch with a small real one. Batch size 1 with 16 accumulation steps has the same optimisation behaviour as batch size 16 and a sixteenth of the activation memory.

Sequence length is the one people forget. Training at 4,096 tokens when your examples are 600 tokens long wastes memory in proportion. Set max_seq_length to your actual data, and use example packing to fill sequences efficiently.

from transformers import BitsAndBytesConfig, TrainingArguments
bnb = BitsAndBytesConfig(
    load_in_4bit=True, bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="bfloat16", bnb_4bit_use_double_quant=True,
)
args = TrainingArguments(
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,      # effective batch 16
    gradient_checkpointing=True,
    optim="paged_adamw_8bit",
    bf16=True, learning_rate=2e-4, num_train_epochs=2,
)

What fits where

16 GB (a free notebook T4 or a consumer card): QLoRA on a 7B or 8B at sequence length 1,024 to 2,048. A 13B is tight and works at short sequence lengths.

24 GB (a 3090 or 4090): QLoRA on a 13B comfortably, a 34B at short sequences, an 8B at long sequence lengths.

48 GB: QLoRA on a 70B at modest sequence length. This is where a single rented card becomes interesting.

80 GB: 70B QLoRA with room, or full fine-tuning of a 7B.

The cost in quality

QLoRA against full fine-tuning loses very little on typical instruction-tuning tasks: the published comparisons find the gap small enough to be inside evaluation noise for most work. The base being 4-bit does cost something on tasks that were already fragile at that quantisation, which by the earlier lesson means long context and lower-resource languages.

If you have the memory, load_in_4bit=False with a 16-bit frozen base and LoRA adapters is slightly better and still cheap. Use QLoRA because the memory is not there, not out of habit.

The practical order when a run will not fit: gradient checkpointing on, sequence length down to your real data, batch size 1 with accumulation, then 4-bit base. Reducing rank is the last thing to try, because it is the only one that reduces what the run can learn.