What a LoRA actually is

The arithmetic that makes it possible

Full fine-tuning updates every weight matrix. For a 7B model that means holding 7 billion gradients and, with a typical optimiser, two further values per parameter, which is why full fine-tuning a 7B model needs something in the region of 70 to 90GB of memory and is out of reach on consumer hardware.

Low-Rank Adaptation makes one substitution. Instead of learning an update to a weight matrix W of shape d x k directly, it learns two much smaller matrices and uses their product:

W_effective = W + (alpha / r) * B @ A

A : r x k      (initialised from a small random distribution)
B : d x r      (initialised to zeros, so training starts as a no-op)

The parameter count drops from d * k to r * (d + k). With d = k = 4096 and r = 16, that is 16.7 million parameters replaced by about 131,000: roughly 0.8% of them. The base weights are frozen, so they need no gradients and no optimiser state, and the memory that full fine-tuning spends on them disappears entirely.

Because B starts at zero, the adapted model is initially identical to the base. Training moves it away from there, which is why LoRA runs are stable from the first step rather than needing a long warm-up to recover from a random perturbation.

Rank, alpha, and the one that actually matters

r is the rank: the width of the bottleneck, and the capacity of the update. alpha is a scaling constant, and the update is multiplied by alpha / r.

That ratio is the part people get wrong. If you raise r from 16 to 64 and leave alpha at 16, you have quartered the scaling and the adapter will appear to learn far less. Common practice is to set alpha equal to r, or to twice r, and to keep the relationship fixed when you change rank so that the effective step size stays comparable.

Rough guidance on rank, to be tested rather than trusted:

• 8 to 16 for style, tone, format and output shape. This covers most real projects.
• 32 to 64 for a genuine new task, a domain vocabulary, or a classification boundary that instructions cannot express.
• 128 and above rarely helps on small datasets, and reliably overfits a few hundred examples.

A higher rank is not a free improvement. It is more capacity to memorise a small dataset, which is exactly the failure mode you are trying to avoid.

Which layers to adapt

The original work adapted only the attention query and value projections. Practice has moved on, and the common configuration now reaches more of the network:

from peft import LoraConfig
cfg = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05, bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj","k_proj","v_proj","o_proj",
                    "gate_proj","up_proj","down_proj"],
)

Including the MLP projections (gate_proj, up_proj, down_proj) costs more parameters and more memory, and it matters for anything beyond surface style, because a great deal of a transformer’s task-specific computation happens there. Attention-only adapters are cheaper and noticeably weaker for real task learning.

Note what is absent: the embedding matrix and the output head. Adapting those is necessary if you are adding tokens to the vocabulary and unnecessary otherwise, and they are large, so leaving them out is most of why adapters are small.

Why it works

The justification is empirical rather than proved. The observation is that the update required to adapt a pretrained model to a downstream task has low intrinsic rank: the change you need lives in a small subspace, even though the weights themselves do not. This holds well across a wide range of adaptation tasks and it is not a theorem.

The limitation follows directly. Where the required change is high-rank (learning a new language, a new script, a domain whose statistics differ from the pretraining corpus) the bottleneck is a real constraint, and no rank you can afford will close the gap. That is the boundary between adaptation and continued pretraining, and it is covered later in this module.

What you end up with

An adapter for a 7B model at rank 16 is typically in the region of 20 to 80MB: small enough to version in a repository, to email, to swap at runtime, and to keep several of alongside one base model in VRAM. That practical property is at least as important as the memory saving during training, and the serving lesson in this module is built on it.