QLoRA, and fine-tuning a 7B model on a free GPU

Combining two ideas

LoRA removed the optimiser memory. The weights themselves are still there: 14 GB for a 7B model at bfloat16, which does not leave room on a 16 GB card for activations and a KV cache.

QLoRA loads the base model in 4-bit and trains LoRA adapters on top of it in higher precision. The frozen base is quantized because nothing updates it; the trainable part stays precise because precision is what it needs. A 7B model then trains in roughly 6 to 8 GB, which fits a free Colab T4 with room to spare.

import torch
from transformers import BitsAndBytesConfig, AutoModelForCausalLM
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct", quantization_config=bnb, device_map="auto"
)
base = prepare_model_for_kbit_training(base, use_gradient_checkpointing=True)
model = get_peft_model(base, LoraConfig(r=16, lora_alpha=32, task_type="CAUSAL_LM",
                                        target_modules="all-linear"))

What each setting is doing

nf4 is a 4-bit data type designed for weights that are roughly normally distributed, which neural network weights are. It preserves more information than plain 4-bit integers at the same size.

bnb_4bit_compute_dtype=torch.bfloat16 means weights are stored in 4 bits and computed in bfloat16: dequantized on the fly for each matrix multiply. This is why QLoRA is slower than bfloat16 training per step even though it uses far less memory. You are trading time for space, deliberately.

use_double_quant quantizes the quantization constants themselves, saving a further few hundred megabytes on a 7B model.

prepare_model_for_kbit_training casts layer norms to float32, enables gradient checkpointing and makes the input embeddings require gradients. Skip it and training is unstable or silently fails to learn: one of those lines whose absence produces a confusing result rather than an error.

What it costs in quality

The published QLoRA results, and most community replications, find results close to 16-bit LoRA on adaptation tasks. “Close” is doing work in that sentence: on tasks where the quantization damage falls (arithmetic, exact structured output, rarely seen languages) the gap is larger, for the mechanism in module three.

The practical stance: use QLoRA when the alternative is not training at all, which for most people on free hardware it is. If you have a card that fits the model at bfloat16, use bfloat16: it is faster and marginally better.

Merging a QLoRA adapter, and the subtlety

You trained against 4-bit weights. Merging the adapter into the full-precision base is standard practice and is not exactly the arithmetic you trained: the adapter learned to correct a quantized base, and it is being applied to an unquantized one. In practice this works and is what almost everybody does, and it is worth evaluating the merged model rather than assuming the numbers carry over.

The cleaner path for serving is to keep the adapter separate and load it over whichever base precision you serve with. It also lets you swap adapters, which merging forfeits.

A realistic budget

On a free Colab T4, a 7B model with QLoRA, 2,000 examples of about 512 tokens, three epochs, batch size 1 with accumulation to 16: roughly two to four hours. On Kaggle’s longer sessions, comfortable. Rented on a 24 GB card at community marketplace rates, a few dollars.

The binding constraint is usually the session length rather than the money, which brings back the module-three discipline: save_steps, push_to_hub, resume_from_checkpoint.

The honest alternative

If you want the result rather than the understanding, AutoTrain on the Hub and libraries like Unsloth and Axolotl wrap all of this behind configuration. Unsloth in particular reports substantial speed and memory improvements through custom kernels, and it is free and open source for single-GPU use. Using them is not cheating. Understanding what they are doing is what this lesson is for, because when a run fails you will need it.

Do this now

Load a 7B model with the QLoRA configuration above on a free GPU and print torch.cuda.max_memory_allocated() after one training step. Compare that against the 56 GB full fine-tuning would have needed. That single comparison is the reason this technique changed who can do this work.