Regularisation that helps, and the kind that does not

Most of it is data, not a knob

In classical machine learning, regularisation is a set of techniques for stopping a model memorising a small dataset. In LLM fine-tuning, the most effective regularisers are not techniques at all: fewer epochs, more data, and a mixture that includes examples outside your narrow task. The knobs below are real, but they are second-order compared to those three.

Say it plainly, because a lot of time is wasted here: if your model is overfitting on 700 examples, lora_dropout will not save you. Another 2,000 examples will.

The knobs that are worth setting

LoRA dropout (0.05 to 0.1). Randomly zeroes elements of the adapter’s input during training. Cheap, standard, mildly helpful on small datasets. At dataset sizes above roughly 10,000 examples people often set it to 0 and lose nothing. Set 0.05 and stop thinking about it.

Weight decay (0 to 0.01). For full fine-tuning, 0.01 is a sensible default. For LoRA it is commonly 0, and the reasoning is that the adapter begins at exactly zero, so decay towards zero is pulling against the only learning happening. Some practitioners use a small value anyway; the effect is minor either way.

Max gradient norm (1.0). Gradient clipping. Not really regularisation: it is insurance. A single pathological example produces an enormous gradient, and without clipping that one step can wreck a model that was training fine. Leave it at 1.0. It costs nothing and it prevents a failure mode that looks like a mystery.

Early stopping: on a task metric. Covered in the previous lesson, and the single most effective item on this list.

The knob that hurts generation

Label smoothing. Instead of a one-hot target, put 0.9 on the correct token and spread 0.1 across the rest. It improves calibration in classifiers and is common practice there.

For generative fine-tuning it is usually a mistake. You are deliberately teaching the model that the correct next token is only 90% correct, which flattens the output distribution; at generation time a flatter distribution means more sampling from the tail, which reads as vagueness and drift. It also makes your loss numbers incomparable with runs that did not use it. Leave it at 0 for text generation, use it if you like on a classification head.

NEFTune, and how to read a claimed win

NEFTune adds uniform random noise to the embedding vectors during training: a few lines of code, one hyperparameter (neftune_noise_alpha, typically 5), and it is built into TRL. The published results showed large gains on AlpacaEval-style benchmarks.

It is worth trying, and it is also a good case study in reading a result. Those benchmarks are judged by a model comparing two responses, and such judges are known to prefer longer, more elaborate answers. NEFTune’s outputs tend to be longer. How much of the measured gain is better quality and how much is length preference is not fully settled, and the honest answer is that some of both is likely.

So: try it, and evaluate it on your metric rather than on the benchmark that made it famous. If your task is extraction into a fixed schema, a technique that makes answers more elaborate is not obviously your friend.

The regularisation that matters most

Mixing general data into your training set. If your 3,000 task examples are all that the model sees, it will drift towards producing only that kind of output. Adding 1,000 examples of ordinary instruction-following (from a permissively licensed public set) keeps the model’s general behaviour anchored while it learns your task.

This is called replay, it is the subject of a lesson in the dataset module, and in measured comparisons it does more to prevent catastrophic forgetting than every knob on this page combined. It is also the one that requires actual work rather than a config line, which is why it is the one most often skipped.

A default block you can copy

lora_dropout       = 0.05
weight_decay       = 0.0      # 0.01 for full fine-tuning
max_grad_norm      = 1.0
label_smoothing    = 0.0      # generation; 0.1 is fine on a classifier head
neftune_noise_alpha= None     # try 5, measure on your own metric

Then spend the time you saved on the dataset. That is not a rhetorical flourish; it is where the difference between a good and a mediocre fine-tune reliably comes from.