{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"lora-mechanics","title":"What a LoRA actually is","url":"https://addaly.com/learn/open-models/lora-mechanics","position":52,"of":71,"minutes":9,"answer":"LoRA freezes the base and trains a low-rank product added to each targeted matrix, so under 1% of parameters carry gradients, and rank.","keyIdea":"LoRA freezes the base and trains a low-rank product added to each targeted matrix, so under 1% of parameters carry gradients, and rank, alpha and which modules are targeted are the three settings that decide whether the run does anything."},"body":{"format":"markdown","text":"## The trick, in one equation\n\nFull fine-tuning updates every weight matrix `W` by some `ΔW` of the same shape. LoRA observes that the useful part of `ΔW` is empirically low-rank, and represents it as a product of two thin matrices:\n\n```\nW' = W + (alpha / r) * B @ A\n\nW : d x k   (frozen, never updated)\nB : d x r   (initialised to zeros)\nA : r x k   (initialised randomly)\nr : the rank, typically 8 to 64\n```\n\nThe base weights never move. Only `A` and `B` are trained. Because `B` starts at zero, the model begins identical to the base and departs from it gradually, which is why LoRA training is stable.\n\n## The arithmetic that makes it cheap\n\nTake one 4096 × 4096 projection. Full tuning updates 16.8 million parameters. At rank 16, LoRA trains 4096 × 16 plus 16 × 4096 = **131,072** parameters, which is 0.8% of the matrix.\n\nAcross a whole 8B model, targeting all the linear layers at rank 16 gives roughly 20 to 40 million trainable parameters, half a percent of the model. Those are the only parameters needing gradients and optimiser state, which is where the memory saving comes from, and it is why a fine-tune that would need several datacentre GPUs fits on a free notebook.\n\n> Figure: The same tune, two budgets. Full fine-tuning an 8B: Every weight updated; Gradients and optimiser state for 8 billion; Around 110 GB of training memory; A 16 GB copy of the model per variant; Several datacentre GPUs. LoRA at rank 16, all linear layers: Base frozen, and B starts at zero; About 40 million trainable, half a per cent; Fits on a free notebook GPU; A 40 to 200 MB adapter per variant; Keep fifty of them. One 4096 by 4096 projection is 16.8 million parameters fully tuned, or 131,072 at rank 16. Print the trainable percentage on every run: if it is far from what you expected, the target modules are wrong and the next three hours will produce nothing.\n\nThe saved adapter is 40 to 200 MB, against 16 GB for a full copy. You can keep fifty of them.\n\n## The three settings that matter\n\n**Rank `r`.** The capacity of the update.\n\n- 8 to 16: format, tone, style, output structure. Most tasks are here.\n- 32 to 64: a new task, a domain with its own vocabulary, a substantial behavioural shift.\n- 128 and above: approaching full fine-tuning in capacity, and usually a sign that continued pretraining is the real requirement.\n\nHigher rank is not safer. It has more capacity to overfit a small dataset and it forgets more.\n\n**Alpha.** A scaling factor: the update is multiplied by `alpha / r`. The convention `alpha = 2 * r` is a reasonable default. What matters is that alpha and rank move together: raising rank while keeping alpha fixed quietly reduces the effective strength of your update, which is a common cause of a tune that seems to do nothing.\n\n**Target modules.** Which matrices get adapters. The original paper adapted only the query and value projections. Later practice found that targeting **all linear layers** (query, key, value, output, and the three feed-forward matrices) works measurably better for the same parameter budget, because most of the model's parameters are in the feed-forward stack.\n\n```python\nfrom peft import LoraConfig, get_peft_model\ncfg = LoraConfig(\n    r=16, lora_alpha=32, lora_dropout=0.05,\n    target_modules=[\"q_proj\",\"k_proj\",\"v_proj\",\"o_proj\",\n                    \"gate_proj\",\"up_proj\",\"down_proj\"],\n    task_type=\"CAUSAL_LM\",\n)\nmodel = get_peft_model(base, cfg)\nmodel.print_trainable_parameters()   # trainable: 41,943,040 || all: 8,072,204,288 || 0.52%\n```\n\nPrint that last line every time. If the trainable percentage is far from what you expected, your target modules are wrong, and the run will produce nothing useful over several hours.\n\n## Merging, or not\n\nAn adapter can be folded into the base weights permanently:\n\n```python\nmerged = model.merge_and_unload()      # W += (alpha/r) B A\nmerged.save_pretrained(\"./my-model\")\n```\n\nMerged models load anywhere and need no adapter support. Unmerged adapters stay tiny, can be swapped at request time, and let one base serve many tunes. The choice belongs to the shipping lesson at the end of this block; for now, keep the adapter, because you can always merge later and cannot easily unmerge.\n\n## Why it works at all\n\nThe honest answer is empirical. The observation is that fine-tuning updates for a specific task have low intrinsic rank: the model is being nudged within a subspace it already occupies rather than being taught something structurally new. This is consistent with the other findings in this block: LoRA is excellent at form and weak at knowledge, because form is a nudge and knowledge is not.\n\nWhere LoRA underperforms full tuning is precisely where you would predict: large amounts of new material, a new language, a new modality. For those, the answer is not a higher rank; it is continued pretraining, several lessons from here."},"check":{"question":"A team raises LoRA rank from 16 to 64 to give the adapter more capacity, leaving lora_alpha at 32. The tuned model changes less than before rather than more. Why?","options":[{"text":"Rank 64 overfits the small dataset, so the model reverts towards the base during training.","correct":false,"tempting":"Overfitting produces worse generalisation, not a smaller departure from the base on training-style inputs."},{"text":"The update is scaled by alpha divided by rank, so holding alpha fixed while raising rank quartered the effective strength of the update.","correct":true},{"text":"Higher rank requires proportionally more data, so with the same dataset the extra parameters stay near zero.","correct":false,"tempting":"Extra parameters do train; the scaling factor rather than data volume explains a systematically weaker effect."},{"text":"Rank above 32 requires targeting all linear layers, and attention-only targeting caps the achievable change.","correct":false,"tempting":"Module choice matters independently, but it was unchanged here, so it cannot explain a change in behaviour."}]},"prev":{"slug":"when-not-to-fine-tune","title":"When not to fine-tune","url":"https://addaly.com/learn/open-models/when-not-to-fine-tune"},"next":{"slug":"qlora-and-memory","title":"Fitting a fine-tune in 16 GB","url":"https://addaly.com/learn/open-models/qlora-and-memory"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What a LoRA actually is', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/lora-mechanics (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What a LoRA actually is","courseTitle":"The Open Model Ecosystem","module":{"index":6,"slug":"adapting-a-model","title":"Adapting a model to your work"},"answer":null,"keyIdea":"LoRA freezes the base and trains a low-rank product added to each targeted matrix, so under 1% of parameters carry gradients, and rank, alpha and which modules are targeted are the three settings that decide whether the run does anything.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"Full fine-tuning an 8B","items":["Every weight updated","Gradients and optimiser state for 8 billion","Around 110 GB of training memory","A 16 GB copy of the model per variant","Several datacentre GPUs"]},"type":"compare","right":{"title":"LoRA at rank 16, all linear layers","items":["Base frozen, and B starts at zero","About 40 million trainable, half a per cent","Fits on a free notebook GPU","A 40 to 200 MB adapter per variant","Keep fifty of them"]},"title":"The same tune, two budgets","caption":"One 4096 by 4096 projection is 16.8 million parameters fully tuned, or 131,072 at rank 16. Print the trainable percentage on every run: if it is far from what you expected, the target modules are wrong and the next three hours will produce nothing."},"alt":"Figure: The same tune, two budgets. Full fine-tuning an 8B: Every weight updated; Gradients and optimiser state for 8 billion; Around 110 GB of training memory; A 16 GB copy of the model per variant; Several datacentre GPUs. LoRA at rank 16, all linear layers: Base frozen, and B starts at zero; About 40 million trainable, half a per cent; Fits on a free notebook GPU; A 40 to 200 MB adapter per variant; Keep fifty of them. One 4096 by 4096 projection is 16.8 million parameters fully tuned, or 131,072 at rank 16. Print the trainable percentage on every run: if it is far from what you expected, the target modules are wrong and the next three hours will produce nothing."}},"blocks":[{"t":"h2","id":"the-trick-in-one-equation","runs":[{"text":"The trick, in one equation"}]},{"t":"p","runs":[{"text":"Full fine-tuning updates every weight matrix "},{"text":"W","code":true},{"text":" by some "},{"text":"ΔW","code":true},{"text":" of the same shape. LoRA observes that the useful part of "},{"text":"ΔW","code":true},{"text":" is empirically low-rank, and represents it as a product of two thin matrices:"}]},{"t":"code","lang":"","label":"Text","text":"W' = W + (alpha / r) * B @ A\n\nW : d x k   (frozen, never updated)\nB : d x r   (initialised to zeros)\nA : r x k   (initialised randomly)\nr : the rank, typically 8 to 64"},{"t":"p","runs":[{"text":"The base weights never move. Only "},{"text":"A","code":true},{"text":" and "},{"text":"B","code":true},{"text":" are trained. Because "},{"text":"B","code":true},{"text":" starts at zero, the model begins identical to the base and departs from it gradually, which is why LoRA training is stable."}]},{"t":"h2","id":"the-arithmetic-that-makes-it-cheap","runs":[{"text":"The arithmetic that makes it cheap"}]},{"t":"p","runs":[{"text":"Take one 4096 × 4096 projection. Full tuning updates 16.8 million parameters. At rank 16, LoRA trains 4096 × 16 plus 16 × 4096 = "},{"text":"131,072","b":true},{"text":" parameters, which is 0.8% of the matrix."}]},{"t":"p","runs":[{"text":"Across a whole 8B model, targeting all the linear layers at rank 16 gives roughly 20 to 40 million trainable parameters, half a percent of the model. Those are the only parameters needing gradients and optimiser state, which is where the memory saving comes from, and it is why a fine-tune that would need several datacentre GPUs fits on a free notebook."}]},{"t":"p","runs":[{"text":"The saved adapter is 40 to 200 MB, against 16 GB for a full copy. You can keep fifty of them."}]},{"t":"h2","id":"the-three-settings-that-matter","runs":[{"text":"The three settings that matter"}]},{"t":"p","runs":[{"text":"Rank ","b":true},{"text":"r","b":true,"code":true},{"text":".","b":true},{"text":" The capacity of the update."}]},{"t":"ul","items":[[{"text":"8 to 16: format, tone, style, output structure. Most tasks are here."}],[{"text":"32 to 64: a new task, a domain with its own vocabulary, a substantial behavioural shift."}],[{"text":"128 and above: approaching full fine-tuning in capacity, and usually a sign that continued pretraining is the real requirement."}]]},{"t":"p","runs":[{"text":"Higher rank is not safer. It has more capacity to overfit a small dataset and it forgets more."}]},{"t":"p","runs":[{"text":"Alpha.","b":true},{"text":" A scaling factor: the update is multiplied by "},{"text":"alpha / r","code":true},{"text":". The convention "},{"text":"alpha = 2 * r","code":true},{"text":" is a reasonable default. What matters is that alpha and rank move together: raising rank while keeping alpha fixed quietly reduces the effective strength of your update, which is a common cause of a tune that seems to do nothing."}]},{"t":"p","runs":[{"text":"Target modules.","b":true},{"text":" Which matrices get adapters. The original paper adapted only the query and value projections. Later practice found that targeting "},{"text":"all linear layers","b":true},{"text":" (query, key, value, output, and the three feed-forward matrices) works measurably better for the same parameter budget, because most of the model’s parameters are in the feed-forward stack."}]},{"t":"code","lang":"python","label":"Python","text":"from peft import LoraConfig, get_peft_model\ncfg = LoraConfig(\n    r=16, lora_alpha=32, lora_dropout=0.05,\n    target_modules=[\"q_proj\",\"k_proj\",\"v_proj\",\"o_proj\",\n                    \"gate_proj\",\"up_proj\",\"down_proj\"],\n    task_type=\"CAUSAL_LM\",\n)\nmodel = get_peft_model(base, cfg)\nmodel.print_trainable_parameters()   # trainable: 41,943,040 || all: 8,072,204,288 || 0.52%"},{"t":"p","runs":[{"text":"Print that last line every time. If the trainable percentage is far from what you expected, your target modules are wrong, and the run will produce nothing useful over several hours."}]},{"t":"h2","id":"merging-or-not","runs":[{"text":"Merging, or not"}]},{"t":"p","runs":[{"text":"An adapter can be folded into the base weights permanently:"}]},{"t":"code","lang":"python","label":"Python","text":"merged = model.merge_and_unload()      # W += (alpha/r) B A\nmerged.save_pretrained(\"./my-model\")"},{"t":"p","runs":[{"text":"Merged models load anywhere and need no adapter support. Unmerged adapters stay tiny, can be swapped at request time, and let one base serve many tunes. The choice belongs to the shipping lesson at the end of this block; for now, keep the adapter, because you can always merge later and cannot easily unmerge."}]},{"t":"h2","id":"why-it-works-at-all","runs":[{"text":"Why it works at all"}]},{"t":"p","runs":[{"text":"The honest answer is empirical. The observation is that fine-tuning updates for a specific task have low intrinsic rank: the model is being nudged within a subspace it already occupies rather than being taught something structurally new. This is consistent with the other findings in this block: LoRA is excellent at form and weak at knowledge, because form is a nudge and knowledge is not."}]},{"t":"p","runs":[{"text":"Where LoRA underperforms full tuning is precisely where you would predict: large amounts of new material, a new language, a new modality. For those, the answer is not a higher rank; it is continued pretraining, several lessons from here."}]}],"check":{"question":[{"text":"A team raises LoRA rank from 16 to 64 to give the adapter more capacity, leaving lora_alpha at 32. The tuned model changes less than before rather than more. Why?"}],"options":[{"text":[{"text":"Rank 64 overfits the small dataset, so the model reverts towards the base during training."}],"correct":false,"tempting":[{"text":"Overfitting produces worse generalisation, not a smaller departure from the base on training-style inputs."}]},{"text":[{"text":"The update is scaled by alpha divided by rank, so holding alpha fixed while raising rank quartered the effective strength of the update."}],"correct":true,"tempting":null},{"text":[{"text":"Higher rank requires proportionally more data, so with the same dataset the extra parameters stay near zero."}],"correct":false,"tempting":[{"text":"Extra parameters do train; the scaling factor rather than data volume explains a systematically weaker effect."}]},{"text":[{"text":"Rank above 32 requires targeting all linear layers, and attention-only targeting caps the achievable change."}],"correct":false,"tempting":[{"text":"Module choice matters independently, but it was unchanged here, so it cannot explain a change in behaviour."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"when-not-to-fine-tune","title":"When not to fine-tune","minutes":8},"next":{"slug":"qlora-and-memory","title":"Fitting a fine-tune in 16 GB","minutes":9}}}