What LoRA is actually doing

A correction, factored

Fine-tuning changes a weight matrix W into W + ΔW. LoRA’s observation is that for adapting a pretrained model to a task, ΔW does not need to be a full matrix. It can be the product of two thin ones.

ΔW = B · A          A is r × k,  B is d × r,  r is small
W' = W + (α/r) · B · A

For a 4,096 × 4,096 projection, W has 16.8 million entries. With r = 16, A has 16 × 4,096 = 65,536 and B has 4,096 × 16 = 65,536, so ΔW is described by 131,072 numbers: 0.78% of the matrix it modifies.

W itself is frozen. It receives no gradient, needs no optimiser state, and can sit in 4-bit quantised form if you like. Only A and B are trained.

The two initialisations, and why they matter

A is initialised with small random values. B is initialised to exactly zero.

That means BA = 0 at step one, so W’ = W: the model at the start of training is bit-for-bit the base model. This is not a detail, it has three consequences you will meet:

• There is no random perturbation to recover from. A LoRA run cannot make the model worse on step one, which full fine-tuning at a bad learning rate certainly can.
• The adapter must travel from nothing to something within the run, which is why LoRA wants a learning rate ten to twenty times higher than full fine-tuning.
• If you initialised both to zero, the gradient of both would be zero forever and nothing would ever train. The asymmetry is load-bearing.

The scaling factor

The α/r term multiplies the adapter’s contribution. The convention exists so that changing r does not silently change the magnitude of the update: double the rank and each individual entry of BA tends to shrink, and dividing by r compensates.

In practice people set α to r or to 2r and treat the pair as one decision. r=16, alpha=32 is the single most common configuration in the wild. The important thing is to know that α and the learning rate push in the same direction (doubling α at a fixed learning rate has an effect very like doubling the learning rate) so do not tune both at once and then wonder which mattered.

Why a low rank is enough

The intuition, which has empirical support and is not a theorem: adapting a pretrained model to a task appears to require far fewer degrees of freedom than the model has parameters. The pretrained weights already contain the representations; the adaptation is a comparatively simple reweighting of what is already there. Measurements of the “intrinsic dimension” of fine-tuning tasks found that surprisingly small subspaces suffice, and LoRA is that observation turned into a method.

The honest flip side: this is exactly why LoRA is poor at adding new capability. A rank-16 correction cannot install a language the base model never saw, or a modality it has no representations for. Low rank is enough when the thing you want is a reweighting, and it is not enough when the thing you want is new machinery. That is the mechanism behind the rule from module one: training teaches shape, not substance.

Seeing it in code

from peft import LoraConfig, get_peft_model

cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                 target_modules=["q_proj","k_proj","v_proj","o_proj",
                                 "gate_proj","up_proj","down_proj"],
                 task_type="CAUSAL_LM")
model = get_peft_model(base, cfg)
model.print_trainable_parameters()
# trainable params: 41,943,040 || all params: 8,072,204,288 || trainable%: 0.5196

Run print_trainable_parameters() on every configuration before you launch. If it prints 0, your target_modules names do not match the architecture and you are about to train nothing for three hours: a failure that produces a perfectly flat, perfectly innocent-looking loss curve.

What you end up with

The artefact on disk is those A and B matrices: roughly 160 MB in bf16 for the 42M-parameter configuration above, against 16 GB for the base model. You can keep a dozen of them for different tasks, swap them at serving time, and email one to a colleague.

At inference you have a choice. Keep them separate and pay a small overhead for the extra matrix multiplications, or merge them: compute W + (α/r)BA once and write out a normal model with no adapter at all. Merging costs nothing at inference and loses the ability to swap. That choice, and what it does to a quantised base, is a lesson of its own later in the course.