{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"regularisation-that-helps","title":"Regularisation that helps, and the kind that does not","url":"https://addaly.com/learn/fine-tuning/regularisation-that-helps","position":18,"of":84,"minutes":8,"answer":"Fewer epochs, more data and a replay mixture regularise a fine-tune far more than any config knob, gradient clipping at 1.0 is cheap insurance against one pathological example.","keyIdea":"Fewer epochs, more data and a replay mixture regularise a fine-tune far more than any config knob, gradient clipping at 1.0 is cheap insurance against one pathological example, and label smoothing flattens the output distribution in a way that harms generation even though it helps classifiers."},"body":{"format":"markdown","text":"## Most of it is data, not a knob\n\nIn classical machine learning, regularisation is a set of techniques for stopping a model memorising a small dataset. In LLM fine-tuning, the most effective regularisers are not techniques at all: fewer epochs, more data, and a mixture that includes examples outside your narrow task. The knobs below are real, but they are second-order compared to those three.\n\nSay it plainly, because a lot of time is wasted here: if your model is overfitting on 700 examples, `lora_dropout` will not save you. Another 2,000 examples will.\n\n## The knobs that are worth setting\n\n**LoRA dropout (0.05 to 0.1).** Randomly zeroes elements of the adapter's input during training. Cheap, standard, mildly helpful on small datasets. At dataset sizes above roughly 10,000 examples people often set it to 0 and lose nothing. Set 0.05 and stop thinking about it.\n\n**Weight decay (0 to 0.01).** For full fine-tuning, 0.01 is a sensible default. For LoRA it is commonly 0, and the reasoning is that the adapter begins at exactly zero, so decay towards zero is pulling against the only learning happening. Some practitioners use a small value anyway; the effect is minor either way.\n\n**Max gradient norm (1.0).** Gradient clipping. Not really regularisation: it is insurance. A single pathological example produces an enormous gradient, and without clipping that one step can wreck a model that was training fine. Leave it at 1.0. It costs nothing and it prevents a failure mode that looks like a mystery.\n\n**Early stopping: on a task metric.** Covered in the previous lesson, and the single most effective item on this list.\n\n## The knob that hurts generation\n\n**Label smoothing.** Instead of a one-hot target, put 0.9 on the correct token and spread 0.1 across the rest. It improves calibration in classifiers and is common practice there.\n\nFor generative fine-tuning it is usually a mistake. You are deliberately teaching the model that the correct next token is only 90% correct, which flattens the output distribution; at generation time a flatter distribution means more sampling from the tail, which reads as vagueness and drift. It also makes your loss numbers incomparable with runs that did not use it. Leave it at 0 for text generation, use it if you like on a classification head.\n\n## NEFTune, and how to read a claimed win\n\nNEFTune adds uniform random noise to the embedding vectors during training: a few lines of code, one hyperparameter (`neftune_noise_alpha`, typically 5), and it is built into TRL. The published results showed large gains on AlpacaEval-style benchmarks.\n\nIt is worth trying, and it is also a good case study in reading a result. Those benchmarks are judged by a model comparing two responses, and such judges are known to prefer longer, more elaborate answers. NEFTune's outputs tend to be longer. How much of the measured gain is better quality and how much is length preference is not fully settled, and the honest answer is that some of both is likely.\n\nSo: try it, and evaluate it on *your* metric rather than on the benchmark that made it famous. If your task is extraction into a fixed schema, a technique that makes answers more elaborate is not obviously your friend.\n\n## The regularisation that matters most\n\n**Mixing general data into your training set.** If your 3,000 task examples are all that the model sees, it will drift towards producing only that kind of output. Adding 1,000 examples of ordinary instruction-following (from a permissively licensed public set) keeps the model's general behaviour anchored while it learns your task.\n\nThis is called replay, it is the subject of a lesson in the dataset module, and in measured comparisons it does more to prevent catastrophic forgetting than every knob on this page combined. It is also the one that requires actual work rather than a config line, which is why it is the one most often skipped.\n\n## A default block you can copy\n\n```python\nlora_dropout       = 0.05\nweight_decay       = 0.0      # 0.01 for full fine-tuning\nmax_grad_norm      = 1.0\nlabel_smoothing    = 0.0      # generation; 0.1 is fine on a classifier head\nneftune_noise_alpha= None     # try 5, measure on your own metric\n```\n\nThen spend the time you saved on the dataset. That is not a rhetorical flourish; it is where the difference between a good and a mediocre fine-tune reliably comes from."},"check":{"question":"Why is label smoothing a poor default for generative fine-tuning although it is standard practice for classifiers?","options":[{"text":"It increases training time substantially, because the loss must be computed against a full distribution rather than a single index.","correct":false,"tempting":"Names a cost that is negligible in practice and would not explain a difference between classification and generation."},{"text":"It conflicts with loss masking, since smoothed targets cannot be combined with ignore_index for prompt tokens.","correct":false,"tempting":"Invents an incompatibility; masking and smoothing operate on different axes and compose without difficulty."},{"text":"Generative models have far larger vocabularies, so spreading 0.1 across 128,000 tokens gives each an effectively zero target and the technique does nothing.","correct":false,"tempting":"Reasons that the effect vanishes at large vocabulary, when the damage comes from the 0.9 placed on the correct token rather than from the spread."},{"text":"It deliberately flattens the target distribution, and at sampling time a flatter distribution draws more from the tail, producing vaguer text.","correct":true}]},"prev":{"slug":"epochs-and-the-loss-curve","title":"Epochs, overfitting and checkpoint selection","url":"https://addaly.com/learn/fine-tuning/epochs-and-the-loss-curve"},"next":{"slug":"memory-arithmetic","title":"Full, LoRA, QLoRA, and the Memory Arithmetic","url":"https://addaly.com/learn/fine-tuning/memory-arithmetic"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Regularisation that helps, and the kind that does not', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/regularisation-that-helps (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Regularisation that helps, and the kind that does not","courseTitle":"Fine-Tuning, and When Not To","module":{"index":2,"slug":"how-training-works","title":"How training actually works"},"answer":null,"keyIdea":"Fewer epochs, more data and a replay mixture regularise a fine-tune far more than any config knob, gradient clipping at 1.0 is cheap insurance against one pathological example, and label smoothing flattens the output distribution in a way that harms generation even though it helps classifiers.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"most-of-it-is-data-not-a-knob","runs":[{"text":"Most of it is data, not a knob"}]},{"t":"p","runs":[{"text":"In classical machine learning, regularisation is a set of techniques for stopping a model memorising a small dataset. In LLM fine-tuning, the most effective regularisers are not techniques at all: fewer epochs, more data, and a mixture that includes examples outside your narrow task. The knobs below are real, but they are second-order compared to those three."}]},{"t":"p","runs":[{"text":"Say it plainly, because a lot of time is wasted here: if your model is overfitting on 700 examples, "},{"text":"lora_dropout","code":true},{"text":" will not save you. Another 2,000 examples will."}]},{"t":"h2","id":"the-knobs-that-are-worth-setting","runs":[{"text":"The knobs that are worth setting"}]},{"t":"p","runs":[{"text":"LoRA dropout (0.05 to 0.1).","b":true},{"text":" Randomly zeroes elements of the adapter’s input during training. Cheap, standard, mildly helpful on small datasets. At dataset sizes above roughly 10,000 examples people often set it to 0 and lose nothing. Set 0.05 and stop thinking about it."}]},{"t":"p","runs":[{"text":"Weight decay (0 to 0.01).","b":true},{"text":" For full fine-tuning, 0.01 is a sensible default. For LoRA it is commonly 0, and the reasoning is that the adapter begins at exactly zero, so decay towards zero is pulling against the only learning happening. Some practitioners use a small value anyway; the effect is minor either way."}]},{"t":"p","runs":[{"text":"Max gradient norm (1.0).","b":true},{"text":" Gradient clipping. Not really regularisation: it is insurance. A single pathological example produces an enormous gradient, and without clipping that one step can wreck a model that was training fine. Leave it at 1.0. It costs nothing and it prevents a failure mode that looks like a mystery."}]},{"t":"p","runs":[{"text":"Early stopping: on a task metric.","b":true},{"text":" Covered in the previous lesson, and the single most effective item on this list."}]},{"t":"h2","id":"the-knob-that-hurts-generation","runs":[{"text":"The knob that hurts generation"}]},{"t":"p","runs":[{"text":"Label smoothing.","b":true},{"text":" Instead of a one-hot target, put 0.9 on the correct token and spread 0.1 across the rest. It improves calibration in classifiers and is common practice there."}]},{"t":"p","runs":[{"text":"For generative fine-tuning it is usually a mistake. You are deliberately teaching the model that the correct next token is only 90% correct, which flattens the output distribution; at generation time a flatter distribution means more sampling from the tail, which reads as vagueness and drift. It also makes your loss numbers incomparable with runs that did not use it. Leave it at 0 for text generation, use it if you like on a classification head."}]},{"t":"h2","id":"neftune-and-how-to-read-a-claimed-win","runs":[{"text":"NEFTune, and how to read a claimed win"}]},{"t":"p","runs":[{"text":"NEFTune adds uniform random noise to the embedding vectors during training: a few lines of code, one hyperparameter ("},{"text":"neftune_noise_alpha","code":true},{"text":", typically 5), and it is built into TRL. The published results showed large gains on AlpacaEval-style benchmarks."}]},{"t":"p","runs":[{"text":"It is worth trying, and it is also a good case study in reading a result. Those benchmarks are judged by a model comparing two responses, and such judges are known to prefer longer, more elaborate answers. NEFTune’s outputs tend to be longer. How much of the measured gain is better quality and how much is length preference is not fully settled, and the honest answer is that some of both is likely."}]},{"t":"p","runs":[{"text":"So: try it, and evaluate it on "},{"text":"your","i":true},{"text":" metric rather than on the benchmark that made it famous. If your task is extraction into a fixed schema, a technique that makes answers more elaborate is not obviously your friend."}]},{"t":"h2","id":"the-regularisation-that-matters-most","runs":[{"text":"The regularisation that matters most"}]},{"t":"p","runs":[{"text":"Mixing general data into your training set.","b":true},{"text":" If your 3,000 task examples are all that the model sees, it will drift towards producing only that kind of output. Adding 1,000 examples of ordinary instruction-following (from a permissively licensed public set) keeps the model’s general behaviour anchored while it learns your task."}]},{"t":"p","runs":[{"text":"This is called replay, it is the subject of a lesson in the dataset module, and in measured comparisons it does more to prevent catastrophic forgetting than every knob on this page combined. It is also the one that requires actual work rather than a config line, which is why it is the one most often skipped."}]},{"t":"h2","id":"a-default-block-you-can-copy","runs":[{"text":"A default block you can copy"}]},{"t":"code","lang":"python","label":"Python","text":"lora_dropout       = 0.05\nweight_decay       = 0.0      # 0.01 for full fine-tuning\nmax_grad_norm      = 1.0\nlabel_smoothing    = 0.0      # generation; 0.1 is fine on a classifier head\nneftune_noise_alpha= None     # try 5, measure on your own metric"},{"t":"p","runs":[{"text":"Then spend the time you saved on the dataset. That is not a rhetorical flourish; it is where the difference between a good and a mediocre fine-tune reliably comes from."}]}],"check":{"question":[{"text":"Why is label smoothing a poor default for generative fine-tuning although it is standard practice for classifiers?"}],"options":[{"text":[{"text":"It increases training time substantially, because the loss must be computed against a full distribution rather than a single index."}],"correct":false,"tempting":[{"text":"Names a cost that is negligible in practice and would not explain a difference between classification and generation."}]},{"text":[{"text":"It conflicts with loss masking, since smoothed targets cannot be combined with ignore_index for prompt tokens."}],"correct":false,"tempting":[{"text":"Invents an incompatibility; masking and smoothing operate on different axes and compose without difficulty."}]},{"text":[{"text":"Generative models have far larger vocabularies, so spreading 0.1 across 128,000 tokens gives each an effectively zero target and the technique does nothing."}],"correct":false,"tempting":[{"text":"Reasons that the effect vanishes at large vocabulary, when the damage comes from the 0.9 placed on the correct token rather than from the spread."}]},{"text":[{"text":"It deliberately flattens the target distribution, and at sampling time a flatter distribution draws more from the tail, producing vaguer text."}],"correct":true,"tempting":null}],"order":[0,2,1,3],"why":null},"prev":{"slug":"epochs-and-the-loss-curve","title":"Epochs, overfitting and checkpoint selection","minutes":9},"next":{"slug":"memory-arithmetic","title":"Full, LoRA, QLoRA, and the Memory Arithmetic","minutes":10}}}