Choosing rank, alpha and which modules to touch

Breadth beats depth

There are two ways to give a LoRA more capacity: raise the rank, or adapt more matrices. The consistent finding across published comparisons and a great deal of practical experience is that which modules you target matters more than the rank you pick.

The early LoRA work adapted only the attention query and value projections, and that convention stuck around long after the evidence moved. On a modern decoder, adapting all seven linear projections at a modest rank beats adapting two at a high rank, at a similar parameter count.

The seven, on a Llama-style architecture:

• Attention: q_proj, k_proj, v_proj, o_proj
• Feed-forward: gate_proj, up_proj, down_proj

The feed-forward matrices are the larger ones (with hidden 4,096 and intermediate 14,336, each MLP projection is 58.7M parameters against the attention projections’ 16.8M), and a good deal of a transformer’s task-specific behaviour appears to live there.

The parameter count, worked

For an 8B model, 32 layers, hidden 4,096, intermediate 14,336, grouped-query attention with 8 key-value heads (so k_proj and v_proj are 4,096 × 1,024), at r = 16:

per layer:  q 131,072  k 81,920  v 81,920  o 131,072
            gate 294,912  up 294,912  down 294,912
            ------------------------------ = 1,310,720
× 32 layers                                = 41,943,040   (0.52% of the model)

attention only (q,k,v,o): 425,984 × 32     = 13,631,488   (0.17%)

Committing those numbers to paper takes two minutes and tells you, before you rent anything, that your adapter file will be about 160 MB and your optimiser state about 340 MB.

Choosing r

A defensible default: r = 16, alpha = 32, all seven modules. From there:

• r = 8 if your dataset is under about 1,000 examples. Less capacity to memorise with.
• r = 32 or 64 for large datasets (tens of thousands of examples), for continued pretraining on a new domain, or when r = 16 visibly plateaus below where you need it.
• r = 128 and above rarely helps on instruction data and starts to inherit full fine-tuning’s problems (more capacity to overfit, more forgetting) without full fine-tuning’s benefits. If you find yourself here, ask whether full fine-tuning is what you actually want.

The test for whether rank is your constraint is cheap: run r = 8 and r = 32 for the same number of steps and compare on your task metric. If they are within noise, rank was never the limiting factor, and it usually is not. Data is.

Alpha, and the one trap

Set alpha = 2r and leave it. The trap is that α and the learning rate are close to interchangeable in effect (the update is scaled by (α/r)·lr) so if you sweep both you are sweeping one thing twice and will draw confident conclusions from a confounded experiment. Fix α = 2r, tune the learning rate.

One genuine exception: at high rank the conventional α/r scaling divides by r, which shrinks the effective update as rank grows and is part of why high ranks often disappoint. Rank-stabilised LoRA (use_rslora=True in PEFT) divides by √r instead, which keeps the update magnitude better behaved. If you are working at r = 64 or above, switch it on; below that it changes little.

Embeddings and the head

Two matrices are usually not adapted and sometimes should be:

• The embedding table and the output head. If you have added new tokens (a new special token, a domain vocabulary) those embeddings start as noise or as an average, and no amount of adapting the transformer blocks fixes them. PEFT’s modules_to_save=["embed_tokens", "lm_head"] trains them fully. This makes your adapter file much larger (an 8B model’s embedding table is 128,256 × 4,096 = 525M parameters, about 1 GB in bf16) so do it only when you have added tokens.
• LayerNorms. Cheap to train, occasionally helpful, rarely decisive.

The shortcut that is usually right

target_modules="all-linear"

PEFT accepts this and resolves it to every linear layer in the model except the output head. It is architecture-agnostic, which means it keeps working when you switch from a Llama-style model to a Qwen or Mistral one whose module names differ. And a list of module names copied from a blog post about a different architecture is one of the more common reasons print_trainable_parameters() reports zero.