LoRA: training 0.1% of the parameters

The observation it rests on

Full fine-tuning updates every weight. For a 7B model that means holding the weights, their gradients, and two optimiser states each: roughly four times the weight memory, so about 56 GB at bfloat16 before activations. No free GPU comes close.

LoRA starts from an empirical observation: the change fine-tuning makes to a weight matrix tends to be low-rank: expressible as the product of two much smaller matrices. So freeze the original weight W and learn a small update instead:

output = W·x + B·A·x, where A is r×d and B is d×r

With r=16 on a 4096×4096 matrix, A and B together hold about 131,000 numbers against the original’s 16.8 million: under one per cent. Across a whole model you typically train 0.1% to 1% of the parameters.

The consequences are what make it matter:

• Optimiser state shrinks by the same factor, which is where the memory actually went.
• The result is a few megabytes, not gigabytes. You can publish dozens of adapters for one base model.
• The base is untouched, so one loaded base model can serve several adapters, swapped per request.
• Quality is usually close to full fine-tuning on adaptation tasks, and the published comparisons bear this out, though not universally, and not for teaching new capabilities.

In code

pip install peft

from peft import LoraConfig, get_peft_model, TaskType

cfg = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    task_type=TaskType.CAUSAL_LM,
)

model = get_peft_model(base_model, cfg)
model.print_trainable_parameters()
# trainable params: 41,943,040 || all params: 7,290,000,000 || trainable%: 0.575

Then hand it to Trainer exactly as before. Nothing else in the loop changes.

The three settings that matter

r: the rank. 8 or 16 for style and format work; 32 to 64 when teaching a new capability or a new domain. Higher r means more capacity and more memory, and past a point it stops helping. Start at 16.

lora_alpha: a scaling factor; the update is multiplied by alpha/r. The common convention is alpha = 2r, so r=16, alpha=32. Changing r without changing alpha silently changes the effective learning rate, which is a confusing way to get inconsistent results.

target_modules: which matrices get adapters. Attention projections only (q_proj, v_proj) is the minimal, cheapest setting. Including the feed-forward projections as well is the modern default and measurably better on harder tasks. You can pass "all-linear" to let PEFT find them, which is convenient and occasionally adapts something you did not intend.

The adapter is a separate thing

model.save_pretrained("my-adapter")      # a few MB

That directory holds adapter_config.json, which names the base model, and adapter_model.safetensors. Loading it:

from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(base_id, dtype=torch.bfloat16,
                                            device_map="auto")
model = PeftModel.from_pretrained(base, "your-name/my-adapter")

The adapter is useless without its exact base, which is why adapter_config.json records it and why the module-one lesson warned about downloading an adapter and wondering why nothing loads.

For deployment you can fold it in permanently:

merged = model.merge_and_unload()
merged.save_pretrained("merged-model")

Merging removes the small inference overhead of the extra matrices and gives an ordinary model that any serving stack loads. It also gives up the ability to swap adapters, and it produces a full-size model you now have to store.

What LoRA is not good at

It adapts what is there. Teaching a model a language it has essentially never seen, or a new capability, is where full fine-tuning or continued pretraining still wins, and where a higher rank is the first thing to try before giving up on adapters.

There are also variants worth knowing exist: DoRA separates magnitude from direction and often does slightly better at the same rank; prefix and prompt tuning train even fewer parameters with less capability. LoRA remains the default because it is well understood, well supported, and good enough.

Do this now

Run print_trainable_parameters() on a LoRA-wrapped model and read the percentage out loud. Then compute what full fine-tuning would have needed in optimiser memory. That ratio is the entire reason this technique exists.