Learning rate, warmup and the schedule

The one hyperparameter that matters most

If you may tune exactly one number, tune this one. Everything else on a fine-tuning config changes results by a few percent. The learning rate changes whether the run works at all.

The values that people actually use cluster tightly, and the clustering has a reason:

• Full fine-tuning of a pretrained LLM: 1e-5 to 2e-5. You are nudging weights that already encode a great deal. Larger values destroy pretrained behaviour quickly: this is the fastest route to catastrophic forgetting there is.
• LoRA: 1e-4 to 3e-4, commonly 2e-4. Ten to twenty times higher, and this is not arbitrary. The adapter starts at exactly zero (B is initialised to zeros, so BA = 0 and the model is unchanged). It has to travel from nothing to something useful in a few thousand steps, and it is a small number of parameters doing it.
• Embedding and classifier heads: 1e-5 to 5e-5, often with a higher rate for the new head than for the pretrained body.

If a LoRA run is not learning, multiply the learning rate by three before you touch anything else. If it is producing gibberish or NaN, divide by three.

Warmup, and what it is protecting against

Starting at the full learning rate on step one is a bad idea for a mechanical reason: AdamW’s v buffer has seen one gradient, so its estimate of each parameter’s typical magnitude is based on a single noisy sample. Dividing by the square root of a bad estimate produces a badly sized step, and an early badly sized step can knock the model somewhere the rest of the run spends its time recovering from.

Warmup ramps the learning rate from near zero to the target over the first few percent of steps, by which time v is a reasonable estimate.

• Typical: warmup_ratio=0.03, or a flat warmup_steps=10 on short runs.
• On runs under 200 steps, ten steps of warmup is 5% of the run and is enough.
• Skipping warmup on a full fine-tune at 2e-5 is usually survivable. Skipping it on a large-batch run, or with fp16, is how you get a loss spike at step 4.

The decay schedule

After warmup the rate comes down. Three choices cover practically everything:

Cosine. Smooth decay from the peak to near zero over the run. The default for good reason: it spends most of the run at a usefully high rate and finishes with small, careful steps that settle the weights. Use it when you know in advance how many steps you will take.

Linear. Straight line down. Marginally worse than cosine in most comparisons, marginally more predictable. Fine.

Constant, or constant with warmup. Necessary when you do not know the length of the run: a job you might extend, or a continued-pretraining stream. It also makes checkpoints comparable across a run, because a cosine-decayed checkpoint at step 500 of a planned 1,000 is a model caught mid-schedule, not a finished one.

The cosine trap worth knowing: if you set a schedule for 1,000 steps and stop at 400, you have not trained a smaller version of the same thing. You have stopped a model whose learning rate was still high, and it will typically be worse than a model trained with a schedule that was set for 400 steps in the first place. Set the schedule length to the run you intend to finish.

Learning rate and batch size move together

A larger batch gives a less noisy gradient, which supports a larger step. The common rules of thumb are to scale the learning rate linearly with batch size, or with its square root; neither is exactly right and both are better than ignoring the relationship.

The practical version: if you double your effective batch size because you added gradient accumulation to fit a longer sequence, and your loss curve goes flatter and slower, raise the learning rate by around 1.4× before concluding the change hurt.

Finding the value cheaply

You do not need a sweep. Run three short jobs (150 steps each, one at 1e-4, one at 2e-4, one at 5e-4) and look at the training loss curve.

• Too low: loss falls slowly and roughly linearly, no elbow.
• About right: a clear fast drop in the first 50 steps, then a smoother decline.
• Too high: an early drop then a rise, visible spikes, or NaN.

On a free T4, three 150-step LoRA runs on a small model is under half an hour. That is the cheapest hour you will spend on the whole project, and it is the one most often skipped in favour of copying a config from a blog post trained on a different dataset at a different batch size.