{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"rank-alpha-and-targets","title":"Choosing rank, alpha and which modules to touch","url":"https://addaly.com/learn/fine-tuning/rank-alpha-and-targets","position":21,"of":84,"minutes":9,"answer":"Adapting all linear projections at r=16 beats adapting the attention pair at a high rank, alpha should be fixed at 2r so it does not confound your learning-rate sweep.","keyIdea":"Adapting all linear projections at r=16 beats adapting the attention pair at a high rank, alpha should be fixed at 2r so it does not confound your learning-rate sweep, and target_modules=\"all-linear\" survives a change of model family where a copied name list does not."},"body":{"format":"markdown","text":"## Breadth beats depth\n\nThere are two ways to give a LoRA more capacity: raise the rank, or adapt more matrices. The consistent finding across published comparisons and a great deal of practical experience is that **which modules you target matters more than the rank you pick**.\n\nThe early LoRA work adapted only the attention query and value projections, and that convention stuck around long after the evidence moved. On a modern decoder, adapting all seven linear projections at a modest rank beats adapting two at a high rank, at a similar parameter count.\n\nThe seven, on a Llama-style architecture:\n\n- Attention: `q_proj`, `k_proj`, `v_proj`, `o_proj`\n- Feed-forward: `gate_proj`, `up_proj`, `down_proj`\n\nThe feed-forward matrices are the larger ones (with hidden 4,096 and intermediate 14,336, each MLP projection is 58.7M parameters against the attention projections' 16.8M), and a good deal of a transformer's task-specific behaviour appears to live there.\n\n## The parameter count, worked\n\nFor an 8B model, 32 layers, hidden 4,096, intermediate 14,336, grouped-query attention with 8 key-value heads (so `k_proj` and `v_proj` are 4,096 × 1,024), at r = 16:\n\n```\nper layer:  q 131,072  k 81,920  v 81,920  o 131,072\n            gate 294,912  up 294,912  down 294,912\n            ------------------------------ = 1,310,720\n× 32 layers                                = 41,943,040   (0.52% of the model)\n\nattention only (q,k,v,o): 425,984 × 32     = 13,631,488   (0.17%)\n```\n\nCommitting those numbers to paper takes two minutes and tells you, before you rent anything, that your adapter file will be about 160 MB and your optimiser state about 340 MB.\n\n## Choosing r\n\nA defensible default: **r = 16, alpha = 32, all seven modules**. From there:\n\n- **r = 8** if your dataset is under about 1,000 examples. Less capacity to memorise with.\n- **r = 32 or 64** for large datasets (tens of thousands of examples), for continued pretraining on a new domain, or when r = 16 visibly plateaus below where you need it.\n- **r = 128 and above** rarely helps on instruction data and starts to inherit full fine-tuning's problems (more capacity to overfit, more forgetting) without full fine-tuning's benefits. If you find yourself here, ask whether full fine-tuning is what you actually want.\n\nThe test for whether rank is your constraint is cheap: run r = 8 and r = 32 for the same number of steps and compare on your task metric. If they are within noise, rank was never the limiting factor, and it usually is not. Data is.\n\n## Alpha, and the one trap\n\nSet `alpha = 2r` and leave it. The trap is that α and the learning rate are close to interchangeable in effect (the update is scaled by `(α/r)·lr`) so if you sweep both you are sweeping one thing twice and will draw confident conclusions from a confounded experiment. Fix α = 2r, tune the learning rate.\n\n> Figure: Trainable parameters on an 8B model at r = 16. Bars (million parameters trained): All seven projections 41.9; Attention only: q, k, v, o 13.6; Query and value only 6.8. Breadth beats depth. Adapting all seven projections at a modest rank beats adapting two at a high rank, and the feed-forward matrices are the larger ones.\n\nOne genuine exception: at high rank the conventional `α/r` scaling divides by r, which shrinks the effective update as rank grows and is part of why high ranks often disappoint. Rank-stabilised LoRA (`use_rslora=True` in PEFT) divides by √r instead, which keeps the update magnitude better behaved. If you are working at r = 64 or above, switch it on; below that it changes little.\n\n## Embeddings and the head\n\nTwo matrices are usually *not* adapted and sometimes should be:\n\n- **The embedding table and the output head.** If you have added new tokens (a new special token, a domain vocabulary) those embeddings start as noise or as an average, and no amount of adapting the transformer blocks fixes them. PEFT's `modules_to_save=[\"embed_tokens\", \"lm_head\"]` trains them fully. This makes your adapter file much larger (an 8B model's embedding table is 128,256 × 4,096 = 525M parameters, about 1 GB in bf16) so do it only when you have added tokens.\n- **LayerNorms.** Cheap to train, occasionally helpful, rarely decisive.\n\n## The shortcut that is usually right\n\n```python\ntarget_modules=\"all-linear\"\n```\n\nPEFT accepts this and resolves it to every linear layer in the model except the output head. It is architecture-agnostic, which means it keeps working when you switch from a Llama-style model to a Qwen or Mistral one whose module names differ. And a list of module names copied from a blog post about a different architecture is one of the more common reasons `print_trainable_parameters()` reports zero."},"check":{"question":"A team gets no improvement from raising LoRA rank from 16 to 64 on a 3,000-example dataset, while switching from attention-only to all seven linear projections at r=16 gives a clear gain. What does this pattern indicate?","options":[{"text":"Rank 64 overfitted the 3,000 examples, so its true capacity gain was cancelled by memorisation.","correct":false,"tempting":"Offers a plausible-sounding mechanism, but overfitting would show as a gain then a decline, not as no change at all."},{"text":"The constraint was not the dimensionality of each correction but which matrices could be corrected at all.","correct":true},{"text":"The alpha was left at 32 while rank rose, so the higher-rank run had a smaller effective update and was undertrained.","correct":false,"tempting":"Identifies a real scaling subtlety but attributes the whole result to it; the all-linear gain at the same alpha and rank is unexplained by this."},{"text":"Grouped-query attention makes the k and v projections small, so attention-only adapters have too few parameters to train stably at any rank.","correct":false,"tempting":"Correctly notes GQA shrinks two matrices but treats parameter count as the issue, when raising rank to 64 would have fixed a pure count problem and did not."}]},"prev":{"slug":"what-lora-is-doing","title":"What LoRA is actually doing","url":"https://addaly.com/learn/fine-tuning/what-lora-is-doing"},"next":{"slug":"qlora-and-nf4","title":"QLoRA: quantising the part you are not training","url":"https://addaly.com/learn/fine-tuning/qlora-and-nf4"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Choosing rank, alpha and which modules to touch', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/rank-alpha-and-targets (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Choosing rank, alpha and which modules to touch","courseTitle":"Fine-Tuning, and When Not To","module":{"index":3,"slug":"the-peft-family","title":"Full, LoRA and the parameter-efficient family"},"answer":null,"keyIdea":"Adapting all linear projections at r=16 beats adapting the attention pair at a high rank, alpha should be fixed at 2r so it does not confound your learning-rate sweep, and target_modules=”all-linear” survives a change of model family where a copied name list does not.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"million parameters trained","title":"Trainable parameters on an 8B model at r = 16","series":[{"label":"All seven projections","value":41.9,"highlight":true},{"label":"Attention only: q, k, v, o","value":13.6},{"label":"Query and value only","value":6.8}],"caption":"Breadth beats depth. Adapting all seven projections at a modest rank beats adapting two at a high rank, and the feed-forward matrices are the larger ones."},"alt":"Figure: Trainable parameters on an 8B model at r = 16. Bars (million parameters trained): All seven projections 41.9; Attention only: q, k, v, o 13.6; Query and value only 6.8. Breadth beats depth. Adapting all seven projections at a modest rank beats adapting two at a high rank, and the feed-forward matrices are the larger ones."}},"blocks":[{"t":"h2","id":"breadth-beats-depth","runs":[{"text":"Breadth beats depth"}]},{"t":"p","runs":[{"text":"There are two ways to give a LoRA more capacity: raise the rank, or adapt more matrices. The consistent finding across published comparisons and a great deal of practical experience is that "},{"text":"which modules you target matters more than the rank you pick","b":true},{"text":"."}]},{"t":"p","runs":[{"text":"The early LoRA work adapted only the attention query and value projections, and that convention stuck around long after the evidence moved. On a modern decoder, adapting all seven linear projections at a modest rank beats adapting two at a high rank, at a similar parameter count."}]},{"t":"p","runs":[{"text":"The seven, on a Llama-style architecture:"}]},{"t":"ul","items":[[{"text":"Attention: "},{"text":"q_proj","code":true},{"text":", "},{"text":"k_proj","code":true},{"text":", "},{"text":"v_proj","code":true},{"text":", "},{"text":"o_proj","code":true}],[{"text":"Feed-forward: "},{"text":"gate_proj","code":true},{"text":", "},{"text":"up_proj","code":true},{"text":", "},{"text":"down_proj","code":true}]]},{"t":"p","runs":[{"text":"The feed-forward matrices are the larger ones (with hidden 4,096 and intermediate 14,336, each MLP projection is 58.7M parameters against the attention projections’ 16.8M), and a good deal of a transformer’s task-specific behaviour appears to live there."}]},{"t":"h2","id":"the-parameter-count-worked","runs":[{"text":"The parameter count, worked"}]},{"t":"p","runs":[{"text":"For an 8B model, 32 layers, hidden 4,096, intermediate 14,336, grouped-query attention with 8 key-value heads (so "},{"text":"k_proj","code":true},{"text":" and "},{"text":"v_proj","code":true},{"text":" are 4,096 × 1,024), at r = 16:"}]},{"t":"code","lang":"","label":"Text","text":"per layer:  q 131,072  k 81,920  v 81,920  o 131,072\n            gate 294,912  up 294,912  down 294,912\n            ------------------------------ = 1,310,720\n× 32 layers                                = 41,943,040   (0.52% of the model)\n\nattention only (q,k,v,o): 425,984 × 32     = 13,631,488   (0.17%)"},{"t":"p","runs":[{"text":"Committing those numbers to paper takes two minutes and tells you, before you rent anything, that your adapter file will be about 160 MB and your optimiser state about 340 MB."}]},{"t":"h2","id":"choosing-r","runs":[{"text":"Choosing r"}]},{"t":"p","runs":[{"text":"A defensible default: "},{"text":"r = 16, alpha = 32, all seven modules","b":true},{"text":". From there:"}]},{"t":"ul","items":[[{"text":"r = 8","b":true},{"text":" if your dataset is under about 1,000 examples. Less capacity to memorise with."}],[{"text":"r = 32 or 64","b":true},{"text":" for large datasets (tens of thousands of examples), for continued pretraining on a new domain, or when r = 16 visibly plateaus below where you need it."}],[{"text":"r = 128 and above","b":true},{"text":" rarely helps on instruction data and starts to inherit full fine-tuning’s problems (more capacity to overfit, more forgetting) without full fine-tuning’s benefits. If you find yourself here, ask whether full fine-tuning is what you actually want."}]]},{"t":"p","runs":[{"text":"The test for whether rank is your constraint is cheap: run r = 8 and r = 32 for the same number of steps and compare on your task metric. If they are within noise, rank was never the limiting factor, and it usually is not. Data is."}]},{"t":"h2","id":"alpha-and-the-one-trap","runs":[{"text":"Alpha, and the one trap"}]},{"t":"p","runs":[{"text":"Set "},{"text":"alpha = 2r","code":true},{"text":" and leave it. The trap is that α and the learning rate are close to interchangeable in effect (the update is scaled by "},{"text":"(α/r)·lr","code":true},{"text":") so if you sweep both you are sweeping one thing twice and will draw confident conclusions from a confounded experiment. Fix α = 2r, tune the learning rate."}]},{"t":"p","runs":[{"text":"One genuine exception: at high rank the conventional "},{"text":"α/r","code":true},{"text":" scaling divides by r, which shrinks the effective update as rank grows and is part of why high ranks often disappoint. Rank-stabilised LoRA ("},{"text":"use_rslora=True","code":true},{"text":" in PEFT) divides by √r instead, which keeps the update magnitude better behaved. If you are working at r = 64 or above, switch it on; below that it changes little."}]},{"t":"h2","id":"embeddings-and-the-head","runs":[{"text":"Embeddings and the head"}]},{"t":"p","runs":[{"text":"Two matrices are usually "},{"text":"not","i":true},{"text":" adapted and sometimes should be:"}]},{"t":"ul","items":[[{"text":"The embedding table and the output head.","b":true},{"text":" If you have added new tokens (a new special token, a domain vocabulary) those embeddings start as noise or as an average, and no amount of adapting the transformer blocks fixes them. PEFT’s "},{"text":"modules_to_save=[\"embed_tokens\", \"lm_head\"]","code":true},{"text":" trains them fully. This makes your adapter file much larger (an 8B model’s embedding table is 128,256 × 4,096 = 525M parameters, about 1 GB in bf16) so do it only when you have added tokens."}],[{"text":"LayerNorms.","b":true},{"text":" Cheap to train, occasionally helpful, rarely decisive."}]]},{"t":"h2","id":"the-shortcut-that-is-usually-right","runs":[{"text":"The shortcut that is usually right"}]},{"t":"code","lang":"python","label":"Python","text":"target_modules=\"all-linear\""},{"t":"p","runs":[{"text":"PEFT accepts this and resolves it to every linear layer in the model except the output head. It is architecture-agnostic, which means it keeps working when you switch from a Llama-style model to a Qwen or Mistral one whose module names differ. And a list of module names copied from a blog post about a different architecture is one of the more common reasons "},{"text":"print_trainable_parameters()","code":true},{"text":" reports zero."}]}],"check":{"question":[{"text":"A team gets no improvement from raising LoRA rank from 16 to 64 on a 3,000-example dataset, while switching from attention-only to all seven linear projections at r=16 gives a clear gain. What does this pattern indicate?"}],"options":[{"text":[{"text":"Rank 64 overfitted the 3,000 examples, so its true capacity gain was cancelled by memorisation."}],"correct":false,"tempting":[{"text":"Offers a plausible-sounding mechanism, but overfitting would show as a gain then a decline, not as no change at all."}]},{"text":[{"text":"The constraint was not the dimensionality of each correction but which matrices could be corrected at all."}],"correct":true,"tempting":null},{"text":[{"text":"The alpha was left at 32 while rank rose, so the higher-rank run had a smaller effective update and was undertrained."}],"correct":false,"tempting":[{"text":"Identifies a real scaling subtlety but attributes the whole result to it; the all-linear gain at the same alpha and rank is unexplained by this."}]},{"text":[{"text":"Grouped-query attention makes the k and v projections small, so attention-only adapters have too few parameters to train stably at any rank."}],"correct":false,"tempting":[{"text":"Correctly notes GQA shrinks two matrices but treats parameter count as the issue, when raising rank to 64 would have fixed a pure count problem and did not."}]}],"order":[2,1,0,3],"why":null},"prev":{"slug":"what-lora-is-doing","title":"What LoRA is actually doing","minutes":9},"next":{"slug":"qlora-and-nf4","title":"QLoRA: quantising the part you are not training","minutes":10}}}