{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"lora-and-peft","title":"LoRA: training 0.1% of the parameters","url":"https://addaly.com/learn/hugging-face/lora-and-peft","position":60,"of":75,"minutes":9,"answer":"LoRA freezes the base weights and learns a low-rank update, so optimiser state shrinks by the same hundredfold factor and the result is a few megabytes tied to one exact base model.","keyIdea":"LoRA freezes the base weights and learns a low-rank update, so optimiser state shrinks by the same hundredfold factor and the result is a few megabytes tied to one exact base model, which is what makes fine-tuning a 7B model on free hardware possible at all."},"body":{"format":"markdown","text":"## The observation it rests on\n\nFull fine-tuning updates every weight. For a 7B model that means holding the weights, their gradients, and two optimiser states each: roughly four times the weight memory, so about 56 GB at bfloat16 before activations. No free GPU comes close.\n\nLoRA starts from an empirical observation: the *change* fine-tuning makes to a weight matrix tends to be low-rank: expressible as the product of two much smaller matrices. So freeze the original weight `W` and learn a small update instead:\n\n> output = W·x + B·A·x, where A is r×d and B is d×r\n\nWith `r=16` on a 4096×4096 matrix, A and B together hold about 131,000 numbers against the original's 16.8 million: under one per cent. Across a whole model you typically train 0.1% to 1% of the parameters.\n\nThe consequences are what make it matter:\n\n> Figure: Training a 7B model: where the memory actually goes. Bars (GB): Full fine-tuning: weights, gradients, two optimiser states 56; LoRA over a bfloat16 base 18; QLoRA over a 4-bit base 7. The weights are close to the same size in the first two rows. What collapses is the optimiser state, which shrinks by the same hundredfold factor as the trainable parameter count. The third row quantizes the frozen base as well, which is what puts a seven-billion-parameter fine-tune on a free Colab card.\n\n- **Optimiser state shrinks by the same factor**, which is where the memory actually went.\n- **The result is a few megabytes**, not gigabytes. You can publish dozens of adapters for one base model.\n- **The base is untouched**, so one loaded base model can serve several adapters, swapped per request.\n- **Quality is usually close to full fine-tuning** on adaptation tasks, and the published comparisons bear this out, though not universally, and not for teaching new capabilities.\n\n## In code\n\n```bash\npip install peft\n```\n\n```python\nfrom peft import LoraConfig, get_peft_model, TaskType\n\ncfg = LoraConfig(\n    r=16,\n    lora_alpha=32,\n    lora_dropout=0.05,\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n                    \"gate_proj\", \"up_proj\", \"down_proj\"],\n    task_type=TaskType.CAUSAL_LM,\n)\n\nmodel = get_peft_model(base_model, cfg)\nmodel.print_trainable_parameters()\n# trainable params: 41,943,040 || all params: 7,290,000,000 || trainable%: 0.575\n```\n\nThen hand it to `Trainer` exactly as before. Nothing else in the loop changes.\n\n## The three settings that matter\n\n**`r`**: the rank. 8 or 16 for style and format work; 32 to 64 when teaching a new capability or a new domain. Higher `r` means more capacity and more memory, and past a point it stops helping. Start at 16.\n\n**`lora_alpha`**: a scaling factor; the update is multiplied by `alpha/r`. The common convention is `alpha = 2r`, so `r=16, alpha=32`. Changing `r` without changing `alpha` silently changes the effective learning rate, which is a confusing way to get inconsistent results.\n\n**`target_modules`**: which matrices get adapters. Attention projections only (`q_proj`, `v_proj`) is the minimal, cheapest setting. Including the feed-forward projections as well is the modern default and measurably better on harder tasks. You can pass `\"all-linear\"` to let PEFT find them, which is convenient and occasionally adapts something you did not intend.\n\n## The adapter is a separate thing\n\n```python\nmodel.save_pretrained(\"my-adapter\")      # a few MB\n```\n\nThat directory holds `adapter_config.json`, which names the base model, and `adapter_model.safetensors`. Loading it:\n\n```python\nfrom peft import PeftModel\n\nbase = AutoModelForCausalLM.from_pretrained(base_id, dtype=torch.bfloat16,\n                                            device_map=\"auto\")\nmodel = PeftModel.from_pretrained(base, \"your-name/my-adapter\")\n```\n\nThe adapter is useless without its exact base, which is why `adapter_config.json` records it and why the module-one lesson warned about downloading an adapter and wondering why nothing loads.\n\nFor deployment you can fold it in permanently:\n\n```python\nmerged = model.merge_and_unload()\nmerged.save_pretrained(\"merged-model\")\n```\n\nMerging removes the small inference overhead of the extra matrices and gives an ordinary model that any serving stack loads. It also gives up the ability to swap adapters, and it produces a full-size model you now have to store.\n\n## What LoRA is not good at\n\nIt adapts what is there. Teaching a model a language it has essentially never seen, or a new capability, is where full fine-tuning or continued pretraining still wins, and where a higher rank is the first thing to try before giving up on adapters.\n\nThere are also variants worth knowing exist: DoRA separates magnitude from direction and often does slightly better at the same rank; prefix and prompt tuning train even fewer parameters with less capability. LoRA remains the default because it is well understood, well supported, and good enough.\n\n## Do this now\n\nRun `print_trainable_parameters()` on a LoRA-wrapped model and read the percentage out loud. Then compute what full fine-tuning would have needed in optimiser memory. That ratio is the entire reason this technique exists."},"check":{"question":"Why does LoRA reduce training memory so much more than it reduces the number of weights involved in the forward pass?","options":[{"text":"Frozen base weights are offloaded to CPU during training, so only the adapter matrices stay resident.","correct":false,"tempting":"Base weights remain on the device because the forward pass needs them every step."},{"text":"Optimiser state and gradients are held only for trainable parameters, and those are a fraction of a per cent.","correct":true},{"text":"The low-rank decomposition reduces the arithmetic in each layer, which shrinks the activations that must be stored.","correct":false,"tempting":"Activations are largely unchanged, since the full base computation still runs alongside the adapter."},{"text":"Adapters are stored in 8-bit while base weights stay in bfloat16, halving the memory for the trained part.","correct":false,"tempting":"Adapter dtype is a minor detail against a hundredfold reduction in the number of optimiser states."}]},"prev":{"slug":"training-arguments-that-matter","title":"The six arguments worth understanding","url":"https://addaly.com/learn/hugging-face/training-arguments-that-matter"},"next":{"slug":"qlora-on-one-card","title":"QLoRA, and fine-tuning a 7B model on a free GPU","url":"https://addaly.com/learn/hugging-face/qlora-on-one-card"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'LoRA: training 0.1% of the parameters', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/lora-and-peft (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"LoRA: training 0.1% of the parameters","courseTitle":"Hugging Face, End to End","module":{"index":7,"slug":"training-with-the-stack","title":"Training and adapting, with the libraries the Hub is built on"},"answer":null,"keyIdea":"LoRA freezes the base weights and learns a low-rank update, so optimiser state shrinks by the same hundredfold factor and the result is a few megabytes tied to one exact base model, which is what makes fine-tuning a 7B model on free hardware possible at all.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"GB","title":"Training a 7B model: where the memory actually goes","series":[{"label":"Full fine-tuning: weights, gradients, two optimiser states","value":56},{"label":"LoRA over a bfloat16 base","value":18},{"label":"QLoRA over a 4-bit base","value":7,"highlight":true}],"caption":"The weights are close to the same size in the first two rows. What collapses is the optimiser state, which shrinks by the same hundredfold factor as the trainable parameter count. The third row quantizes the frozen base as well, which is what puts a seven-billion-parameter fine-tune on a free Colab card."},"alt":"Figure: Training a 7B model: where the memory actually goes. Bars (GB): Full fine-tuning: weights, gradients, two optimiser states 56; LoRA over a bfloat16 base 18; QLoRA over a 4-bit base 7. The weights are close to the same size in the first two rows. What collapses is the optimiser state, which shrinks by the same hundredfold factor as the trainable parameter count. The third row quantizes the frozen base as well, which is what puts a seven-billion-parameter fine-tune on a free Colab card."}},"blocks":[{"t":"h2","id":"the-observation-it-rests-on","runs":[{"text":"The observation it rests on"}]},{"t":"p","runs":[{"text":"Full fine-tuning updates every weight. For a 7B model that means holding the weights, their gradients, and two optimiser states each: roughly four times the weight memory, so about 56 GB at bfloat16 before activations. No free GPU comes close."}]},{"t":"p","runs":[{"text":"LoRA starts from an empirical observation: the "},{"text":"change","i":true},{"text":" fine-tuning makes to a weight matrix tends to be low-rank: expressible as the product of two much smaller matrices. So freeze the original weight "},{"text":"W","code":true},{"text":" and learn a small update instead:"}]},{"t":"quote","runs":[{"text":"output = W·x + B·A·x, where A is r×d and B is d×r"}]},{"t":"p","runs":[{"text":"With "},{"text":"r=16","code":true},{"text":" on a 4096×4096 matrix, A and B together hold about 131,000 numbers against the original’s 16.8 million: under one per cent. Across a whole model you typically train 0.1% to 1% of the parameters."}]},{"t":"p","runs":[{"text":"The consequences are what make it matter:"}]},{"t":"ul","items":[[{"text":"Optimiser state shrinks by the same factor","b":true},{"text":", which is where the memory actually went."}],[{"text":"The result is a few megabytes","b":true},{"text":", not gigabytes. You can publish dozens of adapters for one base model."}],[{"text":"The base is untouched","b":true},{"text":", so one loaded base model can serve several adapters, swapped per request."}],[{"text":"Quality is usually close to full fine-tuning","b":true},{"text":" on adaptation tasks, and the published comparisons bear this out, though not universally, and not for teaching new capabilities."}]]},{"t":"h2","id":"in-code","runs":[{"text":"In code"}]},{"t":"code","lang":"bash","label":"Bash","text":"pip install peft"},{"t":"code","lang":"python","label":"Python","text":"from peft import LoraConfig, get_peft_model, TaskType\n\ncfg = LoraConfig(\n    r=16,\n    lora_alpha=32,\n    lora_dropout=0.05,\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n                    \"gate_proj\", \"up_proj\", \"down_proj\"],\n    task_type=TaskType.CAUSAL_LM,\n)\n\nmodel = get_peft_model(base_model, cfg)\nmodel.print_trainable_parameters()\n# trainable params: 41,943,040 || all params: 7,290,000,000 || trainable%: 0.575"},{"t":"p","runs":[{"text":"Then hand it to "},{"text":"Trainer","code":true},{"text":" exactly as before. Nothing else in the loop changes."}]},{"t":"h2","id":"the-three-settings-that-matter","runs":[{"text":"The three settings that matter"}]},{"t":"p","runs":[{"text":"r","b":true,"code":true},{"text":": the rank. 8 or 16 for style and format work; 32 to 64 when teaching a new capability or a new domain. Higher "},{"text":"r","code":true},{"text":" means more capacity and more memory, and past a point it stops helping. Start at 16."}]},{"t":"p","runs":[{"text":"lora_alpha","b":true,"code":true},{"text":": a scaling factor; the update is multiplied by "},{"text":"alpha/r","code":true},{"text":". The common convention is "},{"text":"alpha = 2r","code":true},{"text":", so "},{"text":"r=16, alpha=32","code":true},{"text":". Changing "},{"text":"r","code":true},{"text":" without changing "},{"text":"alpha","code":true},{"text":" silently changes the effective learning rate, which is a confusing way to get inconsistent results."}]},{"t":"p","runs":[{"text":"target_modules","b":true,"code":true},{"text":": which matrices get adapters. Attention projections only ("},{"text":"q_proj","code":true},{"text":", "},{"text":"v_proj","code":true},{"text":") is the minimal, cheapest setting. Including the feed-forward projections as well is the modern default and measurably better on harder tasks. You can pass "},{"text":"\"all-linear\"","code":true},{"text":" to let PEFT find them, which is convenient and occasionally adapts something you did not intend."}]},{"t":"h2","id":"the-adapter-is-a-separate-thing","runs":[{"text":"The adapter is a separate thing"}]},{"t":"code","lang":"python","label":"Python","text":"model.save_pretrained(\"my-adapter\")      # a few MB"},{"t":"p","runs":[{"text":"That directory holds "},{"text":"adapter_config.json","code":true},{"text":", which names the base model, and "},{"text":"adapter_model.safetensors","code":true},{"text":". Loading it:"}]},{"t":"code","lang":"python","label":"Python","text":"from peft import PeftModel\n\nbase = AutoModelForCausalLM.from_pretrained(base_id, dtype=torch.bfloat16,\n                                            device_map=\"auto\")\nmodel = PeftModel.from_pretrained(base, \"your-name/my-adapter\")"},{"t":"p","runs":[{"text":"The adapter is useless without its exact base, which is why "},{"text":"adapter_config.json","code":true},{"text":" records it and why the module-one lesson warned about downloading an adapter and wondering why nothing loads."}]},{"t":"p","runs":[{"text":"For deployment you can fold it in permanently:"}]},{"t":"code","lang":"python","label":"Python","text":"merged = model.merge_and_unload()\nmerged.save_pretrained(\"merged-model\")"},{"t":"p","runs":[{"text":"Merging removes the small inference overhead of the extra matrices and gives an ordinary model that any serving stack loads. It also gives up the ability to swap adapters, and it produces a full-size model you now have to store."}]},{"t":"h2","id":"what-lora-is-not-good-at","runs":[{"text":"What LoRA is not good at"}]},{"t":"p","runs":[{"text":"It adapts what is there. Teaching a model a language it has essentially never seen, or a new capability, is where full fine-tuning or continued pretraining still wins, and where a higher rank is the first thing to try before giving up on adapters."}]},{"t":"p","runs":[{"text":"There are also variants worth knowing exist: DoRA separates magnitude from direction and often does slightly better at the same rank; prefix and prompt tuning train even fewer parameters with less capability. LoRA remains the default because it is well understood, well supported, and good enough."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Run "},{"text":"print_trainable_parameters()","code":true},{"text":" on a LoRA-wrapped model and read the percentage out loud. Then compute what full fine-tuning would have needed in optimiser memory. That ratio is the entire reason this technique exists."}]}],"check":{"question":[{"text":"Why does LoRA reduce training memory so much more than it reduces the number of weights involved in the forward pass?"}],"options":[{"text":[{"text":"Frozen base weights are offloaded to CPU during training, so only the adapter matrices stay resident."}],"correct":false,"tempting":[{"text":"Base weights remain on the device because the forward pass needs them every step."}]},{"text":[{"text":"Optimiser state and gradients are held only for trainable parameters, and those are a fraction of a per cent."}],"correct":true,"tempting":null},{"text":[{"text":"The low-rank decomposition reduces the arithmetic in each layer, which shrinks the activations that must be stored."}],"correct":false,"tempting":[{"text":"Activations are largely unchanged, since the full base computation still runs alongside the adapter."}]},{"text":[{"text":"Adapters are stored in 8-bit while base weights stay in bfloat16, halving the memory for the trained part."}],"correct":false,"tempting":[{"text":"Adapter dtype is a minor detail against a hundredfold reduction in the number of optimiser states."}]}],"order":[0,1,2,3],"why":null},"prev":{"slug":"training-arguments-that-matter","title":"The six arguments worth understanding","minutes":9},"next":{"slug":"qlora-on-one-card","title":"QLoRA, and fine-tuning a 7B model on a free GPU","minutes":8}}}