QLoRA: training a 7B on the card you have

Where training memory actually goes

Inference needs weights plus a KV cache. Training needs four things, and three of them are invisible until you run out:

• Weights. 7B at 16-bit is about 14GB.
• Gradients. One per trainable parameter, same precision. Another 14GB for a full fine-tune.
• Optimiser state. Adam keeps two moments per parameter, commonly in 32-bit: about 56GB for 7B.
• Activations. Everything the backward pass needs, which scales with batch size and sequence length and is the term people forget.

Add them up and you already have 84GB before activations. Standard mixed-precision training also keeps a 32-bit master copy of the weights, so weights, gradients and optimiser state come to 16 to 18 bytes per parameter: roughly 112 to 126GB for a 7B model before activations. LoRA removes the second and third for the frozen base, because frozen parameters have no gradients and no optimiser state. You are left with 14GB of weights, a few megabytes of adapter gradients and states, and the activations.

QLoRA removes most of the remainder by loading the base in 4-bit. The 14GB becomes roughly 4GB. Training happens in a higher precision, dequantising each weight block as it is needed for the forward and backward pass: the quantised copy is what lives in memory, not what the arithmetic uses.

The practical result is that a 7B model becomes trainable on a 12GB card, and an 8B on a 16GB one, at moderate sequence lengths.

A configuration that fits

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",           # 4-bit NormalFloat, not plain int4
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,      # quantise the quantisation constants too
)
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct", quantization_config=bnb, device_map="auto")
model.gradient_checkpointing_enable()
model.config.use_cache = False           # incompatible with checkpointing

Three of those settings deserve explanation.

NF4 is a 4-bit format whose levels are spaced to match the roughly normal distribution of neural network weights, rather than spaced evenly. It costs the same memory as int4 and loses less. There is no reason to choose otherwise for this.

Double quantisation quantises the per-block scaling constants as well as the weights, saving roughly a further 0.4 bits per parameter. On a 7B model that is a few hundred megabytes, which is often the difference between fitting and not.

Gradient checkpointing discards intermediate activations during the forward pass and recomputes them during the backward pass. It cuts activation memory substantially and costs roughly 20 to 30% in speed. On a card that would otherwise not fit, that trade is not close.

Sequence length is the knob that bites

Activation memory grows with sequence length, and attention’s contribution grows faster than linearly. Halving max_seq_length from 4096 to 2048 typically frees more memory than any other single change, and it is the first thing to try when a run dies in the first few steps.

The cost is that the model never sees examples longer than that during training, which matters if your real inputs are long. The usual compromise: train at the length that covers the great majority of your examples, and accept that behaviour on the tail is untrained rather than learned.

If you must keep the length, reduce per_device_train_batch_size to 1 and recover the effective batch size with gradient_accumulation_steps. Sixteen accumulation steps at batch size 1 gives the same gradient as batch size 16, at the same memory as batch size 1 and roughly the same total time.

Use a paged optimiser

Memory use during training is spiky, and a single spike kills a run that had been fine for two hours. optim="paged_adamw_8bit" keeps the optimiser state in 8-bit and allows it to page to host memory under pressure. It is slightly slower and it converts a class of hard failures into a class of slowdowns.

The honest catch, which arrives at merge time

QLoRA trains an adapter against a base that was quantised to 4-bit. If you then merge that adapter into the original 16-bit weights, the base the adapter was fitted to is not the base it is now attached to. The discrepancy is usually small and it is not always small.

There are two defensible paths. Serve the adapter on the same 4-bit base it was trained against, which is exactly consistent and is what a multi-adapter server does anyway. Or merge into the 16-bit base and then re-run your evaluation, treating the merged model as a new artefact rather than as the thing you tested. What is not defensible is merging and assuming the numbers carry over.

The free path

This entire lesson runs on a free hosted notebook tier. A 7B QLoRA at 2048 tokens on a few thousand examples fits within a free session’s memory and time budget, with the usual caveats: mount your own storage so the adapter survives, checkpoint often, and expect the session to end without warning. For learning this technique, no hardware purchase is required at any point.