The one hyperparameter worth your afternoon

Why this one and not the others

If you can tune exactly one thing, tune the learning rate. It has a larger effect on final quality than the optimiser choice, the initialisation scheme, the batch size or the architecture width, and it is the only hyperparameter whose wrong value can turn a working model into NaN in forty steps.

The reason follows from the previous two lessons. The step you take is learning_rate × gradient. Too large and you overshoot along the sharpest-curvature direction and diverge. Too small and you crawl along the shallowest one. The usable band sits between those, it is usually about one order of magnitude wide, and it moves when you change almost anything else about the model.

The range test: twenty minutes, one plot

There is a cheap way to find the band, and it is worth doing before every serious run.

Start with an absurdly small learning rate, around 1e-7. Train for a few hundred steps, multiplying the learning rate by about 1.1 after each one, and record the loss at each rate. Plot loss against learning rate on a log axis.

lrs, losses = [], []
lr = 1e-7
for batch in itertools.islice(loader, 200):
    for g in opt.param_groups: g['lr'] = lr
    loss = step(batch)
    lrs.append(lr); losses.append(loss)
    lr *= 1.1

The plot has a consistent shape. Flat on the left, where the rate is too small for anything to happen. Then a descent, where the loss falls fastest. Then a sharp rise, where the rate crosses the stability threshold and the run begins to break.

The rate you want is roughly one order of magnitude below where the loss starts rising — near the steepest part of the descent, not at its bottom. Picking the exact minimum of the curve puts you at the edge of instability, and a run that survives 200 steps there will frequently fail at 20,000.

Typical values, and why they are so different

The numbers vary by orders of magnitude across settings, and it helps to know the neighbourhood before you start:

• Training a transformer from scratch with AdamW: 1e-4 to 1e-3.
• Full fine-tuning of a pre-trained model: 1e-5 to 5e-5. Ten to a hundred times smaller, because you are adjusting a good solution rather than searching for one, and a large step destroys what pre-training built.
• LoRA fine-tuning: 1e-4 to 1e-3. Larger again, because the adapter starts from zero and has its own scale.
• SGD with momentum on a vision model: 0.01 to 0.1. Much larger than any Adam figure, because SGD does not divide by the gradient magnitude and Adam effectively does.

If you find yourself needing a rate far outside these bands, the usual cause is a scaling problem in the data or the initialisation rather than a genuinely unusual model.

Warmup, and the reason it exists

Almost every large training run begins with a linear warmup: the learning rate rises from near zero to its target over the first few hundred or few thousand steps.

The mechanism is the one from the Adam lesson. Adam's second-moment estimate v is built from very few samples at the start, so the per-parameter step sizes are based on almost no evidence and can be badly wrong. Bias correction fixes the systematic part but not the variance. Warmup simply avoids taking large steps while the estimates are still noise.

There is a second reason for pre-trained models: the first few batches produce large gradients because the new task's head is random, and a full-size step at that moment can wreck weights that took a great deal of compute to produce.

Decay, and what schedule to use

The rate should fall over training, because the region you are optimising in gets sharper as the loss gets lower. Two schedules cover almost everything:

• Cosine decay from the peak to near zero over the planned number of steps. This is the default for language-model training and works well. Its one demand is that you must know the total number of steps in advance, because the shape depends on it.
• Step decay, dividing by 10 at fixed points. Older, still fine, and easier to adjust mid-run.

A detail that costs people real runs: if you use cosine decay and then decide to train for longer, you cannot simply continue. The schedule has already annealed to near zero, and extending it means the extra steps do almost nothing. Decide the length first.

Batch size interacts with it

Increase the batch size and the gradient estimate becomes less noisy, so a larger step is safe. Two rough rules circulate: scale the learning rate linearly with batch size (common for SGD on vision tasks), or with the square root of the batch size (often better for Adam). Neither is exact and both break down at very large batches.

The practical version: if you change the batch size by more than a factor of two, re-run the range test. Carrying a learning rate across a batch-size change is one of the most common reasons a configuration that worked for somebody else does not work for you.

The rule to keep

Sweep the learning rate before you tune anything else, choose a value about ten times below where the loss starts to rise, warm up into it and decay away from it. Everything else in the training configuration matters less than getting this one number into the right decade.