{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"dora-rslora-and-variants","title":"DoRA, rsLoRA and the variant question","url":"https://addaly.com/learn/fine-tuning/dora-rslora-and-variants","position":23,"of":84,"minutes":9,"answer":"rsLoRA and DoRA fix specific defects (the alpha-over-r damping at high rank and the coupling of magnitude to direction at low rank) but variants are the last term in the sum, behind the dataset.","keyIdea":"rsLoRA and DoRA fix specific defects (the alpha-over-r damping at high rank and the coupling of magnitude to direction at low rank) but variants are the last term in the sum, behind the dataset, the template, the learning rate and the target modules."},"body":{"format":"markdown","text":"## A field with a lot of papers\n\nLoRA has attracted dozens of variants. Most give a point or two on a benchmark, cost some complexity, and do not survive contact with a real dataset any better than the baseline. A few are worth knowing, and knowing *why* they help is more useful than the list.\n\n## rsLoRA: fixing the scaling at high rank\n\nStandard LoRA scales its update by `α/r`. As r grows, that divisor grows linearly, and the effective magnitude of the update falls. This is part of why people report high ranks disappointing: the higher-rank adapter was not just more expressive, it was also more damped.\n\nRank-stabilised LoRA divides by `√r` instead. The update magnitude stays roughly constant as rank rises, so a rank-64 run explores more capacity rather than exploring the same amount more slowly.\n\n```python\nLoraConfig(r=64, lora_alpha=128, use_rslora=True, ...)\n```\n\n**When to use it:** any time r ≥ 32. Below that the difference is small. It costs nothing and it removes a confound from your rank experiments, which is reason enough.\n\n## DoRA: separating magnitude from direction\n\nA weight matrix can be decomposed into a magnitude per column and a direction. DoRA does exactly that, keeps a trainable magnitude vector, and applies the LoRA update to the direction only.\n\nThe motivation is an observed difference in how full fine-tuning and LoRA move weights: full fine-tuning tends to make relatively large directional changes with modest magnitude changes, while LoRA couples the two. Decoupling them makes LoRA's learning dynamics look more like full fine-tuning's.\n\nIn published comparisons DoRA improves over LoRA by a few points at low rank, with the gap narrowing as rank rises, which is consistent with the story, since a higher-rank LoRA can represent the decoupling itself.\n\n```python\nLoraConfig(r=8, lora_alpha=16, use_dora=True, ...)\n```\n\n**The cost:** roughly 20 to 30% slower per step, because of the extra normalisation work, and additional memory for the magnitude vectors. **When to use it:** when you are memory-constrained to a low rank and the extra time is affordable. At r = 32 or above, the case is weak.\n\n## PiSSA and the initialisation family\n\nStandard LoRA starts B at zero, so the adapter begins as a no-op and has to learn from scratch. PiSSA instead initialises A and B from the top singular vectors of the original weight matrix, and freezes the residual. The claim is faster convergence, because the adapter starts aligned with the directions the weight matrix already considers important.\n\nThis family (PiSSA, OLoRA, EVA and others) is interesting and unsettled. The reported gains are real on the benchmarks reported; whether they hold on a specific task with a few thousand examples is the sort of question that only your own ablation answers. Treat them as worth a single comparison run, not as defaults.\n\n## LoRA+ : two learning rates\n\nA and B play asymmetric roles (A maps down to rank r, B maps back up) and the argument is that they should not share a learning rate. LoRA+ gives B a rate several times higher than A, typically 16×. It is a one-line change in some libraries and reports faster convergence at no memory cost.\n\nCheap to try, and it belongs in the same bucket as rsLoRA: a correction to a scaling decision that was made for simplicity rather than for optimality.\n\n## How to decide, without reading fifty papers\n\nA workable policy:\n\n1. **Baseline first.** r = 16, α = 32, all-linear, plain LoRA. Get a number.\n2. **Fix the dataset.** If the baseline is disappointing, the variant is not your problem. Nothing on this page compensates for 600 examples or a broken chat template.\n3. **Then, if you have budget for exactly two more runs:** rsLoRA if you are at high rank, DoRA if you are pinned at low rank. Those are the two with the clearest mechanism and the most replication.\n4. **Measure on your own metric,** not on the paper's benchmark. Most of these methods were evaluated on instruction-following benchmarks scored by a model judge, and your extraction task is not that.\n\n## The uncomfortable summary\n\nThe distribution of outcomes in this field is lopsided. Across a lot of practical work, the ordering of what moves a fine-tune's quality is: the dataset, then the chat template and masking, then the learning rate, then the target modules, then the rank, then the variant. Variants sit last, and a team that reaches for DoRA before checking that its loss mask is correct is optimising the smallest term in the sum."},"check":{"question":"A team finds that raising rank from 16 to 64 gives almost no gain, and that switching on use_rslora at r=64 suddenly does. What was suppressing the higher-rank run?","options":[{"text":"At r=64 the adapter had too many parameters for the dataset, and rsLoRA regularises it back to an effective lower rank.","correct":false,"tempting":"Casts rsLoRA as a regulariser; it changes the update's scaling, and if anything makes the higher-rank adapter move more, not less."},{"text":"Rank 64 exceeded the intrinsic dimension of the task, so the extra directions received no gradient until rsLoRA redistributed it.","correct":false,"tempting":"Treats intrinsic dimension as a hard cap that switches off gradients, when unused directions would simply stay near zero rather than block learning."},{"text":"Higher ranks need more steps to converge, and rsLoRA accelerates convergence by reinitialising the adapter from the weight matrix's singular vectors.","correct":false,"tempting":"Attributes PiSSA's initialisation mechanism to rsLoRA, which changes only the scaling denominator and leaves initialisation untouched."},{"text":"The alpha-over-r scaling divides by rank, so the rank-64 update was damped four times more than the rank-16 one at the same alpha-to-rank ratio.","correct":true}]},"prev":{"slug":"qlora-and-nf4","title":"QLoRA: quantising the part you are not training","url":"https://addaly.com/learn/fine-tuning/qlora-and-nf4"},"next":{"slug":"soft-prompts-and-prefixes","title":"Soft prompts and prefixes","url":"https://addaly.com/learn/fine-tuning/soft-prompts-and-prefixes"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'DoRA, rsLoRA and the variant question', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/dora-rslora-and-variants (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"DoRA, rsLoRA and the variant question","courseTitle":"Fine-Tuning, and When Not To","module":{"index":3,"slug":"the-peft-family","title":"Full, LoRA and the parameter-efficient family"},"answer":null,"keyIdea":"rsLoRA and DoRA fix specific defects (the alpha-over-r damping at high rank and the coupling of magnitude to direction at low rank) but variants are the last term in the sum, behind the dataset, the template, the learning rate and the target modules.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"a-field-with-a-lot-of-papers","runs":[{"text":"A field with a lot of papers"}]},{"t":"p","runs":[{"text":"LoRA has attracted dozens of variants. Most give a point or two on a benchmark, cost some complexity, and do not survive contact with a real dataset any better than the baseline. A few are worth knowing, and knowing "},{"text":"why","i":true},{"text":" they help is more useful than the list."}]},{"t":"h2","id":"rslora-fixing-the-scaling-at-high-rank","runs":[{"text":"rsLoRA: fixing the scaling at high rank"}]},{"t":"p","runs":[{"text":"Standard LoRA scales its update by "},{"text":"α/r","code":true},{"text":". As r grows, that divisor grows linearly, and the effective magnitude of the update falls. This is part of why people report high ranks disappointing: the higher-rank adapter was not just more expressive, it was also more damped."}]},{"t":"p","runs":[{"text":"Rank-stabilised LoRA divides by "},{"text":"√r","code":true},{"text":" instead. The update magnitude stays roughly constant as rank rises, so a rank-64 run explores more capacity rather than exploring the same amount more slowly."}]},{"t":"code","lang":"python","label":"Python","text":"LoraConfig(r=64, lora_alpha=128, use_rslora=True, ...)"},{"t":"p","runs":[{"text":"When to use it:","b":true},{"text":" any time r ≥ 32. Below that the difference is small. It costs nothing and it removes a confound from your rank experiments, which is reason enough."}]},{"t":"h2","id":"dora-separating-magnitude-from-direction","runs":[{"text":"DoRA: separating magnitude from direction"}]},{"t":"p","runs":[{"text":"A weight matrix can be decomposed into a magnitude per column and a direction. DoRA does exactly that, keeps a trainable magnitude vector, and applies the LoRA update to the direction only."}]},{"t":"p","runs":[{"text":"The motivation is an observed difference in how full fine-tuning and LoRA move weights: full fine-tuning tends to make relatively large directional changes with modest magnitude changes, while LoRA couples the two. Decoupling them makes LoRA’s learning dynamics look more like full fine-tuning’s."}]},{"t":"p","runs":[{"text":"In published comparisons DoRA improves over LoRA by a few points at low rank, with the gap narrowing as rank rises, which is consistent with the story, since a higher-rank LoRA can represent the decoupling itself."}]},{"t":"code","lang":"python","label":"Python","text":"LoraConfig(r=8, lora_alpha=16, use_dora=True, ...)"},{"t":"p","runs":[{"text":"The cost:","b":true},{"text":" roughly 20 to 30% slower per step, because of the extra normalisation work, and additional memory for the magnitude vectors. "},{"text":"When to use it:","b":true},{"text":" when you are memory-constrained to a low rank and the extra time is affordable. At r = 32 or above, the case is weak."}]},{"t":"h2","id":"pissa-and-the-initialisation-family","runs":[{"text":"PiSSA and the initialisation family"}]},{"t":"p","runs":[{"text":"Standard LoRA starts B at zero, so the adapter begins as a no-op and has to learn from scratch. PiSSA instead initialises A and B from the top singular vectors of the original weight matrix, and freezes the residual. The claim is faster convergence, because the adapter starts aligned with the directions the weight matrix already considers important."}]},{"t":"p","runs":[{"text":"This family (PiSSA, OLoRA, EVA and others) is interesting and unsettled. The reported gains are real on the benchmarks reported; whether they hold on a specific task with a few thousand examples is the sort of question that only your own ablation answers. Treat them as worth a single comparison run, not as defaults."}]},{"t":"h2","id":"lora-two-learning-rates","runs":[{"text":"LoRA+ : two learning rates"}]},{"t":"p","runs":[{"text":"A and B play asymmetric roles (A maps down to rank r, B maps back up) and the argument is that they should not share a learning rate. LoRA+ gives B a rate several times higher than A, typically 16×. It is a one-line change in some libraries and reports faster convergence at no memory cost."}]},{"t":"p","runs":[{"text":"Cheap to try, and it belongs in the same bucket as rsLoRA: a correction to a scaling decision that was made for simplicity rather than for optimality."}]},{"t":"h2","id":"how-to-decide-without-reading-fifty-papers","runs":[{"text":"How to decide, without reading fifty papers"}]},{"t":"p","runs":[{"text":"A workable policy:"}]},{"t":"ol","start":1,"items":[[{"text":"Baseline first.","b":true},{"text":" r = 16, α = 32, all-linear, plain LoRA. Get a number."}],[{"text":"Fix the dataset.","b":true},{"text":" If the baseline is disappointing, the variant is not your problem. Nothing on this page compensates for 600 examples or a broken chat template."}],[{"text":"Then, if you have budget for exactly two more runs:","b":true},{"text":" rsLoRA if you are at high rank, DoRA if you are pinned at low rank. Those are the two with the clearest mechanism and the most replication."}],[{"text":"Measure on your own metric,","b":true},{"text":" not on the paper’s benchmark. Most of these methods were evaluated on instruction-following benchmarks scored by a model judge, and your extraction task is not that."}]]},{"t":"h2","id":"the-uncomfortable-summary","runs":[{"text":"The uncomfortable summary"}]},{"t":"p","runs":[{"text":"The distribution of outcomes in this field is lopsided. Across a lot of practical work, the ordering of what moves a fine-tune’s quality is: the dataset, then the chat template and masking, then the learning rate, then the target modules, then the rank, then the variant. Variants sit last, and a team that reaches for DoRA before checking that its loss mask is correct is optimising the smallest term in the sum."}]}],"check":{"question":[{"text":"A team finds that raising rank from 16 to 64 gives almost no gain, and that switching on use_rslora at r=64 suddenly does. What was suppressing the higher-rank run?"}],"options":[{"text":[{"text":"At r=64 the adapter had too many parameters for the dataset, and rsLoRA regularises it back to an effective lower rank."}],"correct":false,"tempting":[{"text":"Casts rsLoRA as a regulariser; it changes the update’s scaling, and if anything makes the higher-rank adapter move more, not less."}]},{"text":[{"text":"Rank 64 exceeded the intrinsic dimension of the task, so the extra directions received no gradient until rsLoRA redistributed it."}],"correct":false,"tempting":[{"text":"Treats intrinsic dimension as a hard cap that switches off gradients, when unused directions would simply stay near zero rather than block learning."}]},{"text":[{"text":"Higher ranks need more steps to converge, and rsLoRA accelerates convergence by reinitialising the adapter from the weight matrix’s singular vectors."}],"correct":false,"tempting":[{"text":"Attributes PiSSA’s initialisation mechanism to rsLoRA, which changes only the scaling denominator and leaves initialisation untouched."}]},{"text":[{"text":"The alpha-over-r scaling divides by rank, so the rank-64 update was damped four times more than the rank-16 one at the same alpha-to-rank ratio."}],"correct":true,"tempting":null}],"order":[2,3,1,0],"why":null},"prev":{"slug":"qlora-and-nf4","title":"QLoRA: quantising the part you are not training","minutes":10},"next":{"slug":"soft-prompts-and-prefixes","title":"Soft prompts and prefixes","minutes":8}}}