What a LoRA actually is

The trick, in one equation

Full fine-tuning updates every weight matrix W by some ΔW of the same shape. LoRA observes that the useful part of ΔW is empirically low-rank, and represents it as a product of two thin matrices:

W' = W + (alpha / r) * B @ A

W : d x k   (frozen, never updated)
B : d x r   (initialised to zeros)
A : r x k   (initialised randomly)
r : the rank, typically 8 to 64

The base weights never move. Only A and B are trained. Because B starts at zero, the model begins identical to the base and departs from it gradually, which is why LoRA training is stable.

The arithmetic that makes it cheap

Take one 4096 × 4096 projection. Full tuning updates 16.8 million parameters. At rank 16, LoRA trains 4096 × 16 plus 16 × 4096 = 131,072 parameters, which is 0.8% of the matrix.

Across a whole 8B model, targeting all the linear layers at rank 16 gives roughly 20 to 40 million trainable parameters, half a percent of the model. Those are the only parameters needing gradients and optimiser state, which is where the memory saving comes from, and it is why a fine-tune that would need several datacentre GPUs fits on a free notebook.

The saved adapter is 40 to 200 MB, against 16 GB for a full copy. You can keep fifty of them.

The three settings that matter

Rank r. The capacity of the update.

• 8 to 16: format, tone, style, output structure. Most tasks are here.
• 32 to 64: a new task, a domain with its own vocabulary, a substantial behavioural shift.
• 128 and above: approaching full fine-tuning in capacity, and usually a sign that continued pretraining is the real requirement.

Higher rank is not safer. It has more capacity to overfit a small dataset and it forgets more.

Alpha. A scaling factor: the update is multiplied by alpha / r. The convention alpha = 2 * r is a reasonable default. What matters is that alpha and rank move together: raising rank while keeping alpha fixed quietly reduces the effective strength of your update, which is a common cause of a tune that seems to do nothing.

Target modules. Which matrices get adapters. The original paper adapted only the query and value projections. Later practice found that targeting all linear layers (query, key, value, output, and the three feed-forward matrices) works measurably better for the same parameter budget, because most of the model’s parameters are in the feed-forward stack.

from peft import LoraConfig, get_peft_model
cfg = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=["q_proj","k_proj","v_proj","o_proj",
                    "gate_proj","up_proj","down_proj"],
    task_type="CAUSAL_LM",
)
model = get_peft_model(base, cfg)
model.print_trainable_parameters()   # trainable: 41,943,040 || all: 8,072,204,288 || 0.52%

Print that last line every time. If the trainable percentage is far from what you expected, your target modules are wrong, and the run will produce nothing useful over several hours.

Merging, or not

An adapter can be folded into the base weights permanently:

merged = model.merge_and_unload()      # W += (alpha/r) B A
merged.save_pretrained("./my-model")

Merged models load anywhere and need no adapter support. Unmerged adapters stay tiny, can be swapped at request time, and let one base serve many tunes. The choice belongs to the shipping lesson at the end of this block; for now, keep the adapter, because you can always merge later and cannot easily unmerge.

Why it works at all

The honest answer is empirical. The observation is that fine-tuning updates for a specific task have low intrinsic rank: the model is being nudged within a subspace it already occupies rather than being taught something structurally new. This is consistent with the other findings in this block: LoRA is excellent at form and weak at knowledge, because form is a nudge and knowledge is not.

Where LoRA underperforms full tuning is precisely where you would predict: large amounts of new material, a new language, a new modality. For those, the answer is not a higher rank; it is continued pretraining, several lessons from here.