{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"what-lora-is-doing","title":"What LoRA is actually doing","url":"https://addaly.com/learn/fine-tuning/what-lora-is-doing","position":20,"of":84,"minutes":9,"answer":"LoRA freezes W and learns a rank-r factorisation of the correction, initialised so the adapter starts as an exact no-op, which is why it needs a much higher learning rate.","keyIdea":"LoRA freezes W and learns a rank-r factorisation of the correction, initialised so the adapter starts as an exact no-op, which is why it needs a much higher learning rate, why it cannot be worse than the base at step one, and why it adds reweighting rather than new machinery."},"body":{"format":"markdown","text":"## A correction, factored\n\nFine-tuning changes a weight matrix W into W + ΔW. LoRA's observation is that for adapting a pretrained model to a task, ΔW does not need to be a full matrix. It can be the product of two thin ones.\n\n```\nΔW = B · A          A is r × k,  B is d × r,  r is small\nW' = W + (α/r) · B · A\n```\n\nFor a 4,096 × 4,096 projection, W has 16.8 million entries. With r = 16, A has 16 × 4,096 = 65,536 and B has 4,096 × 16 = 65,536, so ΔW is described by 131,072 numbers: **0.78% of the matrix it modifies**.\n\nW itself is frozen. It receives no gradient, needs no optimiser state, and can sit in 4-bit quantised form if you like. Only A and B are trained.\n\n## The two initialisations, and why they matter\n\nA is initialised with small random values. **B is initialised to exactly zero.**\n\n> Figure: One 4,096 by 4,096 projection, adapted. A two by two grid, rows Full update and LoRA at r = 16, columns Numbers stored and Share of W: Full update and Numbers stored: 16,777,216; Full update and Share of W: 100%; LoRA at r = 16 and Numbers stored: 131,072; LoRA at r = 16 and Share of W: 0.78%. B is initialised to exactly zero, so BA is zero at step one and the model starts as the base model. That is why a LoRA cannot be worse on step one, and why it wants a learning rate ten to twenty times higher.\n\nThat means BA = 0 at step one, so W' = W: the model at the start of training is bit-for-bit the base model. This is not a detail, it has three consequences you will meet:\n\n- There is no random perturbation to recover from. A LoRA run cannot make the model worse on step one, which full fine-tuning at a bad learning rate certainly can.\n- The adapter must travel from nothing to something within the run, which is why LoRA wants a learning rate ten to twenty times higher than full fine-tuning.\n- If you initialised *both* to zero, the gradient of both would be zero forever and nothing would ever train. The asymmetry is load-bearing.\n\n## The scaling factor\n\nThe `α/r` term multiplies the adapter's contribution. The convention exists so that changing r does not silently change the magnitude of the update: double the rank and each individual entry of BA tends to shrink, and dividing by r compensates.\n\nIn practice people set α to r or to 2r and treat the pair as one decision. `r=16, alpha=32` is the single most common configuration in the wild. The important thing is to know that α and the learning rate push in the same direction (doubling α at a fixed learning rate has an effect very like doubling the learning rate) so do not tune both at once and then wonder which mattered.\n\n## Why a low rank is enough\n\nThe intuition, which has empirical support and is not a theorem: adapting a pretrained model to a task appears to require far fewer degrees of freedom than the model has parameters. The pretrained weights already contain the representations; the adaptation is a comparatively simple reweighting of what is already there. Measurements of the \"intrinsic dimension\" of fine-tuning tasks found that surprisingly small subspaces suffice, and LoRA is that observation turned into a method.\n\nThe honest flip side: this is exactly why LoRA is poor at adding new capability. A rank-16 correction cannot install a language the base model never saw, or a modality it has no representations for. Low rank is enough *when the thing you want is a reweighting*, and it is not enough when the thing you want is new machinery. That is the mechanism behind the rule from module one: training teaches shape, not substance.\n\n## Seeing it in code\n\n```python\nfrom peft import LoraConfig, get_peft_model\n\ncfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,\n                 target_modules=[\"q_proj\",\"k_proj\",\"v_proj\",\"o_proj\",\n                                 \"gate_proj\",\"up_proj\",\"down_proj\"],\n                 task_type=\"CAUSAL_LM\")\nmodel = get_peft_model(base, cfg)\nmodel.print_trainable_parameters()\n# trainable params: 41,943,040 || all params: 8,072,204,288 || trainable%: 0.5196\n```\n\nRun `print_trainable_parameters()` on every configuration before you launch. If it prints 0, your `target_modules` names do not match the architecture and you are about to train nothing for three hours: a failure that produces a perfectly flat, perfectly innocent-looking loss curve.\n\n## What you end up with\n\nThe artefact on disk is those A and B matrices: roughly 160 MB in bf16 for the 42M-parameter configuration above, against 16 GB for the base model. You can keep a dozen of them for different tasks, swap them at serving time, and email one to a colleague.\n\nAt inference you have a choice. Keep them separate and pay a small overhead for the extra matrix multiplications, or merge them: compute `W + (α/r)BA` once and write out a normal model with no adapter at all. Merging costs nothing at inference and loses the ability to swap. That choice, and what it does to a quantised base, is a lesson of its own later in the course."},"check":{"question":"Why is B initialised to zeros while A gets small random values, rather than both being randomly initialised?","options":[{"text":"So the adapter is a no-op at step one, while the random A keeps gradients non-zero.","correct":true},{"text":"Because zero-initialised matrices train faster, since their gradients are larger in the early steps.","correct":false,"tempting":"Asserts a property of zero initialisation that does not hold; the motivation is the product being zero, not the gradient being large."},{"text":"Because B is the larger matrix and random values in it would require more memory to store in the optimiser state.","correct":false,"tempting":"Confuses initialisation with storage; optimiser state is allocated per parameter regardless of the values initialised into it."},{"text":"To keep the rank of the product at exactly r, which random initialisation of both matrices would not guarantee.","correct":false,"tempting":"Invokes a rank concern that does not arise; the product of a d×r and an r×k matrix has rank at most r however either is initialised."}]},"prev":{"slug":"memory-arithmetic","title":"Full, LoRA, QLoRA, and the Memory Arithmetic","url":"https://addaly.com/learn/fine-tuning/memory-arithmetic"},"next":{"slug":"rank-alpha-and-targets","title":"Choosing rank, alpha and which modules to touch","url":"https://addaly.com/learn/fine-tuning/rank-alpha-and-targets"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What LoRA is actually doing', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/what-lora-is-doing (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What LoRA is actually doing","courseTitle":"Fine-Tuning, and When Not To","module":{"index":3,"slug":"the-peft-family","title":"Full, LoRA and the parameter-efficient family"},"answer":null,"keyIdea":"LoRA freezes W and learns a rank-r factorisation of the correction, initialised so the adapter starts as an exact no-op, which is why it needs a much higher learning rate, why it cannot be worse than the base at step one, and why it adds reweighting rather than new machinery.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["Numbers stored","Share of W"],"rows":["Full update","LoRA at r = 16"],"type":"matrix","cells":[["16,777,216","100%"],["131,072","0.78%"]],"title":"One 4,096 by 4,096 projection, adapted","caption":"B is initialised to exactly zero, so BA is zero at step one and the model starts as the base model. That is why a LoRA cannot be worse on step one, and why it wants a learning rate ten to twenty times higher.","cellLabels":[["a full matrix","every entry free to move"],["two thin matrices","a reweighting, not new machinery"]]},"alt":"Figure: One 4,096 by 4,096 projection, adapted. A two by two grid, rows Full update and LoRA at r = 16, columns Numbers stored and Share of W: Full update and Numbers stored: 16,777,216; Full update and Share of W: 100%; LoRA at r = 16 and Numbers stored: 131,072; LoRA at r = 16 and Share of W: 0.78%. B is initialised to exactly zero, so BA is zero at step one and the model starts as the base model. That is why a LoRA cannot be worse on step one, and why it wants a learning rate ten to twenty times higher."}},"blocks":[{"t":"h2","id":"a-correction-factored","runs":[{"text":"A correction, factored"}]},{"t":"p","runs":[{"text":"Fine-tuning changes a weight matrix W into W + ΔW. LoRA’s observation is that for adapting a pretrained model to a task, ΔW does not need to be a full matrix. It can be the product of two thin ones."}]},{"t":"code","lang":"","label":"Text","text":"ΔW = B · A          A is r × k,  B is d × r,  r is small\nW' = W + (α/r) · B · A"},{"t":"p","runs":[{"text":"For a 4,096 × 4,096 projection, W has 16.8 million entries. With r = 16, A has 16 × 4,096 = 65,536 and B has 4,096 × 16 = 65,536, so ΔW is described by 131,072 numbers: "},{"text":"0.78% of the matrix it modifies","b":true},{"text":"."}]},{"t":"p","runs":[{"text":"W itself is frozen. It receives no gradient, needs no optimiser state, and can sit in 4-bit quantised form if you like. Only A and B are trained."}]},{"t":"h2","id":"the-two-initialisations-and-why-they-matter","runs":[{"text":"The two initialisations, and why they matter"}]},{"t":"p","runs":[{"text":"A is initialised with small random values. "},{"text":"B is initialised to exactly zero.","b":true}]},{"t":"p","runs":[{"text":"That means BA = 0 at step one, so W’ = W: the model at the start of training is bit-for-bit the base model. This is not a detail, it has three consequences you will meet:"}]},{"t":"ul","items":[[{"text":"There is no random perturbation to recover from. A LoRA run cannot make the model worse on step one, which full fine-tuning at a bad learning rate certainly can."}],[{"text":"The adapter must travel from nothing to something within the run, which is why LoRA wants a learning rate ten to twenty times higher than full fine-tuning."}],[{"text":"If you initialised "},{"text":"both","i":true},{"text":" to zero, the gradient of both would be zero forever and nothing would ever train. The asymmetry is load-bearing."}]]},{"t":"h2","id":"the-scaling-factor","runs":[{"text":"The scaling factor"}]},{"t":"p","runs":[{"text":"The "},{"text":"α/r","code":true},{"text":" term multiplies the adapter’s contribution. The convention exists so that changing r does not silently change the magnitude of the update: double the rank and each individual entry of BA tends to shrink, and dividing by r compensates."}]},{"t":"p","runs":[{"text":"In practice people set α to r or to 2r and treat the pair as one decision. "},{"text":"r=16, alpha=32","code":true},{"text":" is the single most common configuration in the wild. The important thing is to know that α and the learning rate push in the same direction (doubling α at a fixed learning rate has an effect very like doubling the learning rate) so do not tune both at once and then wonder which mattered."}]},{"t":"h2","id":"why-a-low-rank-is-enough","runs":[{"text":"Why a low rank is enough"}]},{"t":"p","runs":[{"text":"The intuition, which has empirical support and is not a theorem: adapting a pretrained model to a task appears to require far fewer degrees of freedom than the model has parameters. The pretrained weights already contain the representations; the adaptation is a comparatively simple reweighting of what is already there. Measurements of the “intrinsic dimension” of fine-tuning tasks found that surprisingly small subspaces suffice, and LoRA is that observation turned into a method."}]},{"t":"p","runs":[{"text":"The honest flip side: this is exactly why LoRA is poor at adding new capability. A rank-16 correction cannot install a language the base model never saw, or a modality it has no representations for. Low rank is enough "},{"text":"when the thing you want is a reweighting","i":true},{"text":", and it is not enough when the thing you want is new machinery. That is the mechanism behind the rule from module one: training teaches shape, not substance."}]},{"t":"h2","id":"seeing-it-in-code","runs":[{"text":"Seeing it in code"}]},{"t":"code","lang":"python","label":"Python","text":"from peft import LoraConfig, get_peft_model\n\ncfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,\n                 target_modules=[\"q_proj\",\"k_proj\",\"v_proj\",\"o_proj\",\n                                 \"gate_proj\",\"up_proj\",\"down_proj\"],\n                 task_type=\"CAUSAL_LM\")\nmodel = get_peft_model(base, cfg)\nmodel.print_trainable_parameters()\n# trainable params: 41,943,040 || all params: 8,072,204,288 || trainable%: 0.5196"},{"t":"p","runs":[{"text":"Run "},{"text":"print_trainable_parameters()","code":true},{"text":" on every configuration before you launch. If it prints 0, your "},{"text":"target_modules","code":true},{"text":" names do not match the architecture and you are about to train nothing for three hours: a failure that produces a perfectly flat, perfectly innocent-looking loss curve."}]},{"t":"h2","id":"what-you-end-up-with","runs":[{"text":"What you end up with"}]},{"t":"p","runs":[{"text":"The artefact on disk is those A and B matrices: roughly 160 MB in bf16 for the 42M-parameter configuration above, against 16 GB for the base model. You can keep a dozen of them for different tasks, swap them at serving time, and email one to a colleague."}]},{"t":"p","runs":[{"text":"At inference you have a choice. Keep them separate and pay a small overhead for the extra matrix multiplications, or merge them: compute "},{"text":"W + (α/r)BA","code":true},{"text":" once and write out a normal model with no adapter at all. Merging costs nothing at inference and loses the ability to swap. That choice, and what it does to a quantised base, is a lesson of its own later in the course."}]}],"check":{"question":[{"text":"Why is B initialised to zeros while A gets small random values, rather than both being randomly initialised?"}],"options":[{"text":[{"text":"So the adapter is a no-op at step one, while the random A keeps gradients non-zero."}],"correct":true,"tempting":null},{"text":[{"text":"Because zero-initialised matrices train faster, since their gradients are larger in the early steps."}],"correct":false,"tempting":[{"text":"Asserts a property of zero initialisation that does not hold; the motivation is the product being zero, not the gradient being large."}]},{"text":[{"text":"Because B is the larger matrix and random values in it would require more memory to store in the optimiser state."}],"correct":false,"tempting":[{"text":"Confuses initialisation with storage; optimiser state is allocated per parameter regardless of the values initialised into it."}]},{"text":[{"text":"To keep the rank of the product at exactly r, which random initialisation of both matrices would not guarantee."}],"correct":false,"tempting":[{"text":"Invokes a rank concern that does not arise; the product of a d×r and an r×k matrix has rank at most r however either is initialised."}]}],"order":[2,1,3,0],"why":null},"prev":{"slug":"memory-arithmetic","title":"Full, LoRA, QLoRA, and the Memory Arithmetic","minutes":10},"next":{"slug":"rank-alpha-and-targets","title":"Choosing rank, alpha and which modules to touch","minutes":9}}}