Full, LoRA, QLoRA, and the Memory Arithmetic

Count the bytes

Every training setup holds four things in memory: weights, gradients, optimizer state, and activations. Only the last depends on batch size, which is why “just lower the batch size” so often fails to save you.

For a model with P parameters, standard mixed-precision training with AdamW costs roughly:

A 7B model: 7e9 x 16 = 112 GB, before a single activation. That does not fit on one 80 GB card. Full fine-tuning of a 7B is a multi-GPU job with FSDP or DeepSpeed, or an 8-bit-optimizer-plus-offload arrangement that is slow and fiddly. This is why almost nobody does it, and why the rest of this lesson exists.

LoRA: freeze the base, train a small correction

LoRA freezes W and learns a low-rank update, W + BA. For a weight of shape d_in x d_out, B and A hold r(d_in + d_out) parameters. A 4096 x 4096 projection at r=16 is 16 x 8192 = 131,072 parameters against 16.7M: about 0.8%. Across a 7B model with adapters on every linear layer at r=16, you train roughly 20-40M parameters.

The 16-byte tax now applies only to those. The frozen base still sits in memory at 2 bytes per parameter.

• 7B base in bf16: 14 GB
• adapters, their gradients and optimizer state: under 1 GB
• activations with gradient checkpointing, sequence 1024, small batch: 2-5 GB

Call it 18-20 GB. That fits a 24 GB card (RTX 3090, 4090, A10G). It does not fit a 16 GB T4.

QLoRA: quantize the frozen base

The base is frozen, so it does not need to be precise: it only has to produce good forward activations. QLoRA stores it in 4-bit NF4, about 0.5 bytes per parameter plus quantization constants, and dequantizes each block on the fly during the forward pass. Adapters stay in 16-bit.

• 7B base in NF4: about 4 GB
• everything else as before: 3-6 GB

Seven to ten gigabytes total. That fits a free Colab T4, a Kaggle P100, or a 12 GB consumer card. The cost is speed: expect 20-40% lower throughput than 16-bit LoRA because of the dequantization work.

from transformers import BitsAndBytesConfig
bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="bfloat16",  # use float16 on a T4; Turing has no bf16
    bnb_4bit_use_double_quant=True,     # quantizes the quantization constants too
)

Scaling to your model

Rough rule: QLoRA needs about 0.6 x P bytes for the base, plus 3-6 GB of working memory.

• 3B → about 2 GB base → comfortable on 8 GB
• 7-8B → about 4.5 GB → fits 16 GB
• 13B → about 8 GB → tight on 16 GB, fine on 24 GB
• 70B → about 40 GB → needs one 48 GB or 80 GB card

If you have 8 GB of RAM and no GPU

The honest thing: you cannot usefully fine-tune a 7B model on a CPU. A step that takes 0.4 seconds on a rented GPU takes minutes, and a run that takes twenty minutes takes a fortnight.

What you can do, and it is not a consolation prize:

• Fine-tune a small model locally. A 135M-600M parameter model trains on CPU in hours on a few hundred examples. Every part of the pipeline (chat templates, loss masking, evaluation) is identical to the 7B version, and those are the parts where mistakes actually happen.
• Use free GPU hours. Kaggle gives roughly 30 GPU-hours a week on a T4 or P100, with sessions that survive better than Colab’s free tier. Colab free works but disconnects; checkpoint every few hundred steps and push somewhere persistent.
• Rent. A 24 GB card runs $0.30-0.80 an hour on the spot-style marketplaces. Most LoRA runs in this course cost less than a coffee. Prices move constantly; check on the day.

The memory table is the thing to remember. Batch size touches one row of it.