{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"what-a-lora-actually-is","title":"What a LoRA actually is","url":"https://addaly.com/learn/running-models-yourself/what-a-lora-actually-is","position":75,"of":82,"minutes":9,"answer":"A LoRA replaces a full weight update with the product of two thin matrices.","keyIdea":"A LoRA replaces a full weight update with the product of two thin matrices, so only a fraction of a per cent of the parameters train and the frozen base needs no optimiser state. Rank sets capacity, the alpha-over-rank ratio sets effective step size, and adapting the MLP projections as well as attention is what makes it learn tasks rather than only style."},"body":{"format":"markdown","text":"## The arithmetic that makes it possible\n\nFull fine-tuning updates every weight matrix. For a 7B model that means holding 7 billion gradients and, with a typical optimiser, two further values per parameter, which is why full fine-tuning a 7B model needs something in the region of 70 to 90GB of memory and is out of reach on consumer hardware.\n\nLow-Rank Adaptation makes one substitution. Instead of learning an update to a weight matrix `W` of shape `d x k` directly, it learns two much smaller matrices and uses their product:\n\n```\nW_effective = W + (alpha / r) * B @ A\n\nA : r x k      (initialised from a small random distribution)\nB : d x r      (initialised to zeros, so training starts as a no-op)\n```\n\nThe parameter count drops from `d * k` to `r * (d + k)`. With `d = k = 4096` and `r = 16`, that is 16.7 million parameters replaced by about 131,000: roughly 0.8% of them. The base weights are frozen, so they need no gradients and no optimiser state, and the memory that full fine-tuning spends on them disappears entirely.\n\nBecause `B` starts at zero, the adapted model is initially identical to the base. Training moves it away from there, which is why LoRA runs are stable from the first step rather than needing a long warm-up to recover from a random perturbation.\n\n## Rank, alpha, and the one that actually matters\n\n`r` is the rank: the width of the bottleneck, and the capacity of the update. `alpha` is a scaling constant, and the update is multiplied by `alpha / r`.\n\nThat ratio is the part people get wrong. If you raise `r` from 16 to 64 and leave `alpha` at 16, you have quartered the scaling and the adapter will appear to learn far less. Common practice is to set `alpha` equal to `r`, or to twice `r`, and to keep the relationship fixed when you change rank so that the effective step size stays comparable.\n\nRough guidance on rank, to be tested rather than trusted:\n\n- **8 to 16** for style, tone, format and output shape. This covers most real projects.\n- **32 to 64** for a genuine new task, a domain vocabulary, or a classification boundary that instructions cannot express.\n- **128 and above** rarely helps on small datasets, and reliably overfits a few hundred examples.\n\nA higher rank is not a free improvement. It is more capacity to memorise a small dataset, which is exactly the failure mode you are trying to avoid.\n\n## Which layers to adapt\n\nThe original work adapted only the attention query and value projections. Practice has moved on, and the common configuration now reaches more of the network:\n\n```python\nfrom peft import LoraConfig\ncfg = LoraConfig(\n    r=16, lora_alpha=32, lora_dropout=0.05, bias=\"none\",\n    task_type=\"CAUSAL_LM\",\n    target_modules=[\"q_proj\",\"k_proj\",\"v_proj\",\"o_proj\",\n                    \"gate_proj\",\"up_proj\",\"down_proj\"],\n)\n```\n\nIncluding the MLP projections (`gate_proj`, `up_proj`, `down_proj`) costs more parameters and more memory, and it matters for anything beyond surface style, because a great deal of a transformer's task-specific computation happens there. Attention-only adapters are cheaper and noticeably weaker for real task learning.\n\nNote what is absent: the embedding matrix and the output head. Adapting those is necessary if you are adding tokens to the vocabulary and unnecessary otherwise, and they are large, so leaving them out is most of why adapters are small.\n\n## Why it works\n\nThe justification is empirical rather than proved. The observation is that the update required to adapt a pretrained model to a downstream task has low intrinsic rank: the change you need lives in a small subspace, even though the weights themselves do not. This holds well across a wide range of adaptation tasks and it is not a theorem.\n\nThe limitation follows directly. Where the required change is high-rank (learning a new language, a new script, a domain whose statistics differ from the pretraining corpus) the bottleneck is a real constraint, and no rank you can afford will close the gap. That is the boundary between adaptation and continued pretraining, and it is covered later in this module.\n\n## What you end up with\n\nAn adapter for a 7B model at rank 16 is typically in the region of 20 to 80MB: small enough to version in a repository, to email, to swap at runtime, and to keep several of alongside one base model in VRAM. That practical property is at least as important as the memory saving during training, and the serving lesson in this module is built on it."},"check":{"question":"Someone raises LoRA rank from 16 to 64 to give an adapter more capacity, leaving alpha at 16, and finds the trained adapter barely changes the model's behaviour. What is the mechanism?","options":[{"text":"Higher ranks need proportionally more training data, so 64 is undertrained on the same dataset.","correct":false,"tempting":"Undertrained high ranks usually show as overfitting or noise rather than as a uniform reduction in effect across all outputs."},{"text":"The update is scaled by alpha divided by rank, so quadrupling rank quartered its contribution.","correct":true},{"text":"Rank 64 exceeds the intrinsic rank of the required update, so the extra directions learn nothing and dilute the rest.","correct":false,"tempting":"Extra directions may contribute little, but they do not shrink the contribution of the useful ones; the scaling factor does."},{"text":"Matrix B is initialised to zeros, and larger ranks take proportionally longer to move away from that initialisation.","correct":false,"tempting":"The zero initialisation applies at any rank and is escaped within the first optimiser steps; it does not scale with the width of the bottleneck."}]},"prev":{"slug":"prompt-retrieval-or-training","title":"Prompt, retrieval or training: choosing the right tool","url":"https://addaly.com/learn/running-models-yourself/prompt-retrieval-or-training"},"next":{"slug":"qlora-on-one-card","title":"QLoRA: training a 7B on the card you have","url":"https://addaly.com/learn/running-models-yourself/qlora-on-one-card"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What a LoRA actually is', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/what-a-lora-actually-is (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What a LoRA actually is","courseTitle":"Running Models Yourself","module":{"index":9,"slug":"making-it-yours","title":"Making it yours: adapters and training"},"answer":null,"keyIdea":"A LoRA replaces a full weight update with the product of two thin matrices, so only a fraction of a per cent of the parameters train and the frozen base needs no optimiser state. Rank sets capacity, the alpha-over-rank ratio sets effective step size, and adapting the MLP projections as well as attention is what makes it learn tasks rather than only style.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-arithmetic-that-makes-it-possible","runs":[{"text":"The arithmetic that makes it possible"}]},{"t":"p","runs":[{"text":"Full fine-tuning updates every weight matrix. For a 7B model that means holding 7 billion gradients and, with a typical optimiser, two further values per parameter, which is why full fine-tuning a 7B model needs something in the region of 70 to 90GB of memory and is out of reach on consumer hardware."}]},{"t":"p","runs":[{"text":"Low-Rank Adaptation makes one substitution. Instead of learning an update to a weight matrix "},{"text":"W","code":true},{"text":" of shape "},{"text":"d x k","code":true},{"text":" directly, it learns two much smaller matrices and uses their product:"}]},{"t":"code","lang":"","label":"Text","text":"W_effective = W + (alpha / r) * B @ A\n\nA : r x k      (initialised from a small random distribution)\nB : d x r      (initialised to zeros, so training starts as a no-op)"},{"t":"p","runs":[{"text":"The parameter count drops from "},{"text":"d * k","code":true},{"text":" to "},{"text":"r * (d + k)","code":true},{"text":". With "},{"text":"d = k = 4096","code":true},{"text":" and "},{"text":"r = 16","code":true},{"text":", that is 16.7 million parameters replaced by about 131,000: roughly 0.8% of them. The base weights are frozen, so they need no gradients and no optimiser state, and the memory that full fine-tuning spends on them disappears entirely."}]},{"t":"p","runs":[{"text":"Because "},{"text":"B","code":true},{"text":" starts at zero, the adapted model is initially identical to the base. Training moves it away from there, which is why LoRA runs are stable from the first step rather than needing a long warm-up to recover from a random perturbation."}]},{"t":"h2","id":"rank-alpha-and-the-one-that-actually-matters","runs":[{"text":"Rank, alpha, and the one that actually matters"}]},{"t":"p","runs":[{"text":"r","code":true},{"text":" is the rank: the width of the bottleneck, and the capacity of the update. "},{"text":"alpha","code":true},{"text":" is a scaling constant, and the update is multiplied by "},{"text":"alpha / r","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"That ratio is the part people get wrong. If you raise "},{"text":"r","code":true},{"text":" from 16 to 64 and leave "},{"text":"alpha","code":true},{"text":" at 16, you have quartered the scaling and the adapter will appear to learn far less. Common practice is to set "},{"text":"alpha","code":true},{"text":" equal to "},{"text":"r","code":true},{"text":", or to twice "},{"text":"r","code":true},{"text":", and to keep the relationship fixed when you change rank so that the effective step size stays comparable."}]},{"t":"p","runs":[{"text":"Rough guidance on rank, to be tested rather than trusted:"}]},{"t":"ul","items":[[{"text":"8 to 16","b":true},{"text":" for style, tone, format and output shape. This covers most real projects."}],[{"text":"32 to 64","b":true},{"text":" for a genuine new task, a domain vocabulary, or a classification boundary that instructions cannot express."}],[{"text":"128 and above","b":true},{"text":" rarely helps on small datasets, and reliably overfits a few hundred examples."}]]},{"t":"p","runs":[{"text":"A higher rank is not a free improvement. It is more capacity to memorise a small dataset, which is exactly the failure mode you are trying to avoid."}]},{"t":"h2","id":"which-layers-to-adapt","runs":[{"text":"Which layers to adapt"}]},{"t":"p","runs":[{"text":"The original work adapted only the attention query and value projections. Practice has moved on, and the common configuration now reaches more of the network:"}]},{"t":"code","lang":"python","label":"Python","text":"from peft import LoraConfig\ncfg = LoraConfig(\n    r=16, lora_alpha=32, lora_dropout=0.05, bias=\"none\",\n    task_type=\"CAUSAL_LM\",\n    target_modules=[\"q_proj\",\"k_proj\",\"v_proj\",\"o_proj\",\n                    \"gate_proj\",\"up_proj\",\"down_proj\"],\n)"},{"t":"p","runs":[{"text":"Including the MLP projections ("},{"text":"gate_proj","code":true},{"text":", "},{"text":"up_proj","code":true},{"text":", "},{"text":"down_proj","code":true},{"text":") costs more parameters and more memory, and it matters for anything beyond surface style, because a great deal of a transformer’s task-specific computation happens there. Attention-only adapters are cheaper and noticeably weaker for real task learning."}]},{"t":"p","runs":[{"text":"Note what is absent: the embedding matrix and the output head. Adapting those is necessary if you are adding tokens to the vocabulary and unnecessary otherwise, and they are large, so leaving them out is most of why adapters are small."}]},{"t":"h2","id":"why-it-works","runs":[{"text":"Why it works"}]},{"t":"p","runs":[{"text":"The justification is empirical rather than proved. The observation is that the update required to adapt a pretrained model to a downstream task has low intrinsic rank: the change you need lives in a small subspace, even though the weights themselves do not. This holds well across a wide range of adaptation tasks and it is not a theorem."}]},{"t":"p","runs":[{"text":"The limitation follows directly. Where the required change is high-rank (learning a new language, a new script, a domain whose statistics differ from the pretraining corpus) the bottleneck is a real constraint, and no rank you can afford will close the gap. That is the boundary between adaptation and continued pretraining, and it is covered later in this module."}]},{"t":"h2","id":"what-you-end-up-with","runs":[{"text":"What you end up with"}]},{"t":"p","runs":[{"text":"An adapter for a 7B model at rank 16 is typically in the region of 20 to 80MB: small enough to version in a repository, to email, to swap at runtime, and to keep several of alongside one base model in VRAM. That practical property is at least as important as the memory saving during training, and the serving lesson in this module is built on it."}]}],"check":{"question":[{"text":"Someone raises LoRA rank from 16 to 64 to give an adapter more capacity, leaving alpha at 16, and finds the trained adapter barely changes the model’s behaviour. What is the mechanism?"}],"options":[{"text":[{"text":"Higher ranks need proportionally more training data, so 64 is undertrained on the same dataset."}],"correct":false,"tempting":[{"text":"Undertrained high ranks usually show as overfitting or noise rather than as a uniform reduction in effect across all outputs."}]},{"text":[{"text":"The update is scaled by alpha divided by rank, so quadrupling rank quartered its contribution."}],"correct":true,"tempting":null},{"text":[{"text":"Rank 64 exceeds the intrinsic rank of the required update, so the extra directions learn nothing and dilute the rest."}],"correct":false,"tempting":[{"text":"Extra directions may contribute little, but they do not shrink the contribution of the useful ones; the scaling factor does."}]},{"text":[{"text":"Matrix B is initialised to zeros, and larger ranks take proportionally longer to move away from that initialisation."}],"correct":false,"tempting":[{"text":"The zero initialisation applies at any rank and is escaped within the first optimiser steps; it does not scale with the width of the bottleneck."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"prompt-retrieval-or-training","title":"Prompt, retrieval or training: choosing the right tool","minutes":9},"next":{"slug":"qlora-on-one-card","title":"QLoRA: training a 7B on the card you have","minutes":10}}}