{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"learning-rate-and-schedules","title":"Learning rate, warmup and the schedule","url":"https://addaly.com/learn/fine-tuning/learning-rate-and-schedules","position":13,"of":84,"minutes":9,"answer":"LoRA needs a learning rate ten to twenty times higher than full fine-tuning because its adapter starts at exactly zero, warmup exists to stop AdamW taking a large step from a one-sample variance estimate.","keyIdea":"LoRA needs a learning rate ten to twenty times higher than full fine-tuning because its adapter starts at exactly zero, warmup exists to stop AdamW taking a large step from a one-sample variance estimate, and a cosine schedule must be set for the run you actually intend to finish."},"body":{"format":"markdown","text":"## The one hyperparameter that matters most\n\nIf you may tune exactly one number, tune this one. Everything else on a fine-tuning config changes results by a few percent. The learning rate changes whether the run works at all.\n\nThe values that people actually use cluster tightly, and the clustering has a reason:\n\n- **Full fine-tuning of a pretrained LLM: 1e-5 to 2e-5.** You are nudging weights that already encode a great deal. Larger values destroy pretrained behaviour quickly: this is the fastest route to catastrophic forgetting there is.\n- **LoRA: 1e-4 to 3e-4, commonly 2e-4.** Ten to twenty times higher, and this is not arbitrary. The adapter starts at exactly zero (B is initialised to zeros, so BA = 0 and the model is unchanged). It has to travel from nothing to something useful in a few thousand steps, and it is a small number of parameters doing it.\n- **Embedding and classifier heads: 1e-5 to 5e-5**, often with a higher rate for the new head than for the pretrained body.\n\nIf a LoRA run is not learning, multiply the learning rate by three before you touch anything else. If it is producing gibberish or NaN, divide by three.\n\n## Warmup, and what it is protecting against\n\nStarting at the full learning rate on step one is a bad idea for a mechanical reason: AdamW's v buffer has seen one gradient, so its estimate of each parameter's typical magnitude is based on a single noisy sample. Dividing by the square root of a bad estimate produces a badly sized step, and an early badly sized step can knock the model somewhere the rest of the run spends its time recovering from.\n\nWarmup ramps the learning rate from near zero to the target over the first few percent of steps, by which time v is a reasonable estimate.\n\n- Typical: `warmup_ratio=0.03`, or a flat `warmup_steps=10` on short runs.\n- On runs under 200 steps, ten steps of warmup is 5% of the run and is enough.\n- Skipping warmup on a full fine-tune at 2e-5 is usually survivable. Skipping it on a large-batch run, or with fp16, is how you get a loss spike at step 4.\n\n## The decay schedule\n\nAfter warmup the rate comes down. Three choices cover practically everything:\n\n**Cosine.** Smooth decay from the peak to near zero over the run. The default for good reason: it spends most of the run at a usefully high rate and finishes with small, careful steps that settle the weights. Use it when you know in advance how many steps you will take.\n\n**Linear.** Straight line down. Marginally worse than cosine in most comparisons, marginally more predictable. Fine.\n\n**Constant, or constant with warmup.** Necessary when you do not know the length of the run: a job you might extend, or a continued-pretraining stream. It also makes checkpoints comparable across a run, because a cosine-decayed checkpoint at step 500 of a planned 1,000 is a model caught mid-schedule, not a finished one.\n\nThe cosine trap worth knowing: if you set a schedule for 1,000 steps and stop at 400, you have not trained a smaller version of the same thing. You have stopped a model whose learning rate was still high, and it will typically be worse than a model trained with a schedule that was set for 400 steps in the first place. **Set the schedule length to the run you intend to finish.**\n\n## Learning rate and batch size move together\n\nA larger batch gives a less noisy gradient, which supports a larger step. The common rules of thumb are to scale the learning rate linearly with batch size, or with its square root; neither is exactly right and both are better than ignoring the relationship.\n\nThe practical version: if you double your effective batch size because you added gradient accumulation to fit a longer sequence, and your loss curve goes flatter and slower, raise the learning rate by around 1.4× before concluding the change hurt.\n\n## Finding the value cheaply\n\nYou do not need a sweep. Run three short jobs (150 steps each, one at 1e-4, one at 2e-4, one at 5e-4) and look at the training loss curve.\n\n- Too low: loss falls slowly and roughly linearly, no elbow.\n- About right: a clear fast drop in the first 50 steps, then a smoother decline.\n- Too high: an early drop then a rise, visible spikes, or NaN.\n\nOn a free T4, three 150-step LoRA runs on a small model is under half an hour. That is the cheapest hour you will spend on the whole project, and it is the one most often skipped in favour of copying a config from a blog post trained on a different dataset at a different batch size."},"check":{"question":"A team plans a 1,000-step LoRA run with a cosine schedule, then stops it at step 400 to save money and ships that checkpoint. It performs worse than a colleague's run that was configured for 400 steps from the start. Why?","options":[{"text":"The stopped run saw the same data in a different order, and cosine schedules are sensitive to data ordering.","correct":false,"tempting":"Attributes the gap to ordering noise, which would be symmetric and would not reliably favour the shorter-configured run."},{"text":"Checkpoints saved mid-schedule omit the optimiser state, so the weights are incompletely updated when reloaded.","correct":false,"tempting":"Confuses resuming a run with evaluating a checkpoint; optimiser state is irrelevant to the quality of weights already written."},{"text":"At step 400 of a 1,000-step cosine the rate is still high, so the weights never got the settling steps.","correct":true},{"text":"Stopping early truncates warmup, so the model never reached its intended peak learning rate.","correct":false,"tempting":"Inverts the timeline: warmup completes in the first few percent of steps, long before step 400."}]},"prev":{"slug":"optimisers-and-their-memory","title":"Optimisers, and what their state costs","url":"https://addaly.com/learn/fine-tuning/optimisers-and-their-memory"},"next":{"slug":"batch-size-and-accumulation","title":"Batch size, accumulation and the token count","url":"https://addaly.com/learn/fine-tuning/batch-size-and-accumulation"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Learning rate, warmup and the schedule', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/learning-rate-and-schedules (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Learning rate, warmup and the schedule","courseTitle":"Fine-Tuning, and When Not To","module":{"index":2,"slug":"how-training-works","title":"How training actually works"},"answer":null,"keyIdea":"LoRA needs a learning rate ten to twenty times higher than full fine-tuning because its adapter starts at exactly zero, warmup exists to stop AdamW taking a large step from a one-sample variance estimate, and a cosine schedule must be set for the run you actually intend to finish.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-one-hyperparameter-that-matters-most","runs":[{"text":"The one hyperparameter that matters most"}]},{"t":"p","runs":[{"text":"If you may tune exactly one number, tune this one. Everything else on a fine-tuning config changes results by a few percent. The learning rate changes whether the run works at all."}]},{"t":"p","runs":[{"text":"The values that people actually use cluster tightly, and the clustering has a reason:"}]},{"t":"ul","items":[[{"text":"Full fine-tuning of a pretrained LLM: 1e-5 to 2e-5.","b":true},{"text":" You are nudging weights that already encode a great deal. Larger values destroy pretrained behaviour quickly: this is the fastest route to catastrophic forgetting there is."}],[{"text":"LoRA: 1e-4 to 3e-4, commonly 2e-4.","b":true},{"text":" Ten to twenty times higher, and this is not arbitrary. The adapter starts at exactly zero (B is initialised to zeros, so BA = 0 and the model is unchanged). It has to travel from nothing to something useful in a few thousand steps, and it is a small number of parameters doing it."}],[{"text":"Embedding and classifier heads: 1e-5 to 5e-5","b":true},{"text":", often with a higher rate for the new head than for the pretrained body."}]]},{"t":"p","runs":[{"text":"If a LoRA run is not learning, multiply the learning rate by three before you touch anything else. If it is producing gibberish or NaN, divide by three."}]},{"t":"h2","id":"warmup-and-what-it-is-protecting-against","runs":[{"text":"Warmup, and what it is protecting against"}]},{"t":"p","runs":[{"text":"Starting at the full learning rate on step one is a bad idea for a mechanical reason: AdamW’s v buffer has seen one gradient, so its estimate of each parameter’s typical magnitude is based on a single noisy sample. Dividing by the square root of a bad estimate produces a badly sized step, and an early badly sized step can knock the model somewhere the rest of the run spends its time recovering from."}]},{"t":"p","runs":[{"text":"Warmup ramps the learning rate from near zero to the target over the first few percent of steps, by which time v is a reasonable estimate."}]},{"t":"ul","items":[[{"text":"Typical: "},{"text":"warmup_ratio=0.03","code":true},{"text":", or a flat "},{"text":"warmup_steps=10","code":true},{"text":" on short runs."}],[{"text":"On runs under 200 steps, ten steps of warmup is 5% of the run and is enough."}],[{"text":"Skipping warmup on a full fine-tune at 2e-5 is usually survivable. Skipping it on a large-batch run, or with fp16, is how you get a loss spike at step 4."}]]},{"t":"h2","id":"the-decay-schedule","runs":[{"text":"The decay schedule"}]},{"t":"p","runs":[{"text":"After warmup the rate comes down. Three choices cover practically everything:"}]},{"t":"p","runs":[{"text":"Cosine.","b":true},{"text":" Smooth decay from the peak to near zero over the run. The default for good reason: it spends most of the run at a usefully high rate and finishes with small, careful steps that settle the weights. Use it when you know in advance how many steps you will take."}]},{"t":"p","runs":[{"text":"Linear.","b":true},{"text":" Straight line down. Marginally worse than cosine in most comparisons, marginally more predictable. Fine."}]},{"t":"p","runs":[{"text":"Constant, or constant with warmup.","b":true},{"text":" Necessary when you do not know the length of the run: a job you might extend, or a continued-pretraining stream. It also makes checkpoints comparable across a run, because a cosine-decayed checkpoint at step 500 of a planned 1,000 is a model caught mid-schedule, not a finished one."}]},{"t":"p","runs":[{"text":"The cosine trap worth knowing: if you set a schedule for 1,000 steps and stop at 400, you have not trained a smaller version of the same thing. You have stopped a model whose learning rate was still high, and it will typically be worse than a model trained with a schedule that was set for 400 steps in the first place. "},{"text":"Set the schedule length to the run you intend to finish.","b":true}]},{"t":"h2","id":"learning-rate-and-batch-size-move-together","runs":[{"text":"Learning rate and batch size move together"}]},{"t":"p","runs":[{"text":"A larger batch gives a less noisy gradient, which supports a larger step. The common rules of thumb are to scale the learning rate linearly with batch size, or with its square root; neither is exactly right and both are better than ignoring the relationship."}]},{"t":"p","runs":[{"text":"The practical version: if you double your effective batch size because you added gradient accumulation to fit a longer sequence, and your loss curve goes flatter and slower, raise the learning rate by around 1.4× before concluding the change hurt."}]},{"t":"h2","id":"finding-the-value-cheaply","runs":[{"text":"Finding the value cheaply"}]},{"t":"p","runs":[{"text":"You do not need a sweep. Run three short jobs (150 steps each, one at 1e-4, one at 2e-4, one at 5e-4) and look at the training loss curve."}]},{"t":"ul","items":[[{"text":"Too low: loss falls slowly and roughly linearly, no elbow."}],[{"text":"About right: a clear fast drop in the first 50 steps, then a smoother decline."}],[{"text":"Too high: an early drop then a rise, visible spikes, or NaN."}]]},{"t":"p","runs":[{"text":"On a free T4, three 150-step LoRA runs on a small model is under half an hour. That is the cheapest hour you will spend on the whole project, and it is the one most often skipped in favour of copying a config from a blog post trained on a different dataset at a different batch size."}]}],"check":{"question":[{"text":"A team plans a 1,000-step LoRA run with a cosine schedule, then stops it at step 400 to save money and ships that checkpoint. It performs worse than a colleague’s run that was configured for 400 steps from the start. Why?"}],"options":[{"text":[{"text":"The stopped run saw the same data in a different order, and cosine schedules are sensitive to data ordering."}],"correct":false,"tempting":[{"text":"Attributes the gap to ordering noise, which would be symmetric and would not reliably favour the shorter-configured run."}]},{"text":[{"text":"Checkpoints saved mid-schedule omit the optimiser state, so the weights are incompletely updated when reloaded."}],"correct":false,"tempting":[{"text":"Confuses resuming a run with evaluating a checkpoint; optimiser state is irrelevant to the quality of weights already written."}]},{"text":[{"text":"At step 400 of a 1,000-step cosine the rate is still high, so the weights never got the settling steps."}],"correct":true,"tempting":null},{"text":[{"text":"Stopping early truncates warmup, so the model never reached its intended peak learning rate."}],"correct":false,"tempting":[{"text":"Inverts the timeline: warmup completes in the first few percent of steps, long before step 400."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"optimisers-and-their-memory","title":"Optimisers, and what their state costs","minutes":9},"next":{"slug":"batch-size-and-accumulation","title":"Batch size, accumulation and the token count","minutes":9}}}