{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"optimisers-beyond-sgd","title":"Momentum, Adam, and what the optimiser is doing for you","url":"https://addaly.com/learn/machine-learning-foundations/optimisers-beyond-sgd","position":78,"of":92,"minutes":8,"answer":"Momentum averages recent gradients so oscillations cancel and consistent directions accumulate; Adam adds a per-parameter step size from the average squared gradient, and its weight decay must be applied outside that scaling.","keyIdea":"Momentum averages recent gradients so oscillations cancel and consistent directions accumulate; Adam adds a per-parameter step size from the average squared gradient, and its weight decay must be applied outside that scaling, which is what AdamW does."},"body":{"format":"markdown","text":"## What plain SGD struggles with\n\nGradient descent follows the slope where it currently stands. Two situations make that inefficient.\n\n**Ravines.** A surface that is steep across and shallow along: the long narrow valley from module 2. The gradient points mostly across the valley, so the optimiser zigzags from wall to wall and makes slow progress along the floor.\n\n**Uneven scales.** One parameter needs large steps and another needs small ones, and a single learning rate has to serve both.\n\n## Momentum\n\nKeep a running average of recent gradients and step in that direction.\n\n```text\nv = beta * v + gradient          # beta typically 0.9\nw = w - learning_rate * v\n```\n\nThe zigzag components across the ravine alternate in sign and cancel in the average. The consistent component along the floor accumulates. The result is faster progress in the useful direction and damped oscillation, and it also carries the optimiser through small bumps and flat regions where the raw gradient is near zero.\n\nWith `beta = 0.9`, the effective step is roughly ten times the current gradient once the average has built up, which is why momentum with a given learning rate is considerably more aggressive than plain SGD with the same rate.\n\n**Nesterov momentum** evaluates the gradient at the position the momentum is about to take you rather than where you are: a lookahead that damps overshoot slightly. A small, real improvement.\n\n## Adam\n\nAdam adds a per-parameter learning rate on top of momentum. It keeps two running averages: one of the gradient, and one of the gradient *squared*. Then it divides the step by the square root of the second.\n\n```text\nm = b1 * m + (1 - b1) * g          # average gradient\nv = b2 * v + (1 - b2) * g * g      # average squared gradient\nw = w - lr * m / (sqrt(v) + eps)\n```\n\nThe effect: a parameter that consistently receives large gradients has a large `v`, so its steps are scaled down; one receiving small gradients gets steps scaled up. Every parameter ends up with a step size suited to its own history, which removes most of the sensitivity to feature scaling and to how the layers happen to be sized.\n\nDefaults `b1 = 0.9`, `b2 = 0.999`, `lr = 0.001` work across an enormous range of problems, which is why Adam is what almost everyone uses by default.\n\n## Adam's costs\n\n**Memory.** Two extra values per parameter. For a model with a billion parameters, the optimiser state is twice the size of the model, and it is a large part of why training needs so much more memory than inference.\n\n**Generalisation.** On image classification specifically, well-tuned SGD with momentum often generalises slightly better than Adam. The gap is small and the tuning effort is not, and Adam remains the right default when you do not have time to tune.\n\n**Weight decay is broken in plain Adam.** L2 regularisation added to the loss gets divided by the same `sqrt(v)` term, so parameters with large gradients receive weaker decay, which is the opposite of intended. **AdamW** applies the decay directly to the weights, outside the adaptive scaling, and it is what you should use.\n\n```python\ntorch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)\n```\n\nAdamW is the standard for training transformers and should be your default over Adam. The fix is one letter and it matters.\n\n## Schedules\n\nThe learning rate should not stay constant, for the reasons in module 2.\n\n**Warmup**: increase from near zero over the first few hundred to few thousand steps. Essential for transformers, where early gradients on freshly initialised weights are unreliable and a large early step does lasting damage.\n\n**Cosine decay**: fall smoothly to near zero over training. The current standard.\n\n**One-cycle**: warm up to a high rate, then decay below the starting point. Trains surprisingly fast and is built into fastai and PyTorch.\n\n```python\nsched = torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=1e-3,\n                                            total_steps=n_steps)\n```\n\n## Gradient clipping\n\nOne more safety measure. If the gradient's norm exceeds a threshold, scale it down.\n\n```python\ntorch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n```\n\nThis prevents a single unusual batch from taking a catastrophic step and destroying weights that took hours to train. It is standard for recurrent networks and transformers and costs essentially nothing. If your loss occasionally spikes to `NaN` after training normally for a while, this is usually the fix.\n\n## What to actually use\n\nAdamW at 3e-4, cosine decay with warmup, gradient clipping at 1.0. That combination trains almost anything without drama. Reach for SGD with momentum only if you are chasing the last fraction of a point on a vision benchmark and have the compute to tune it."},"check":{"question":"A team uses Adam with weight_decay=0.01 and finds regularisation appears weaker than expected on the parameters that change most. Why does switching to AdamW help?","options":[{"text":"Adam's momentum term cancels the decay over successive steps, while AdamW omits momentum entirely.","correct":false,"tempting":"AdamW retains momentum; removing it would change the optimiser's behaviour far more than the observed regularisation difference."},{"text":"AdamW uses a larger effective decay coefficient by default, so the same value regularises more strongly.","correct":false,"tempting":"The coefficient's meaning differs between the two, but the fix is where the decay is applied rather than a change in its magnitude."},{"text":"AdamW disables the adaptive per-parameter scaling, so every weight receives the same decay.","correct":false,"tempting":"AdamW keeps the adaptive step sizes for the gradient term; only the decay is moved outside them."},{"text":"In Adam the L2 term enters the loss and is then divided by the square root of the average squared gradient, so parameters with large gradients get weaker decay; AdamW applies the decay directly to the weights, outside that scaling.","correct":true}]},"prev":{"slug":"initialisation-and-normalisation","title":"Starting well, and staying well-behaved","url":"https://addaly.com/learn/machine-learning-foundations/initialisation-and-normalisation"},"next":{"slug":"regularising-a-network","title":"Keeping a large network from memorising","url":"https://addaly.com/learn/machine-learning-foundations/regularising-a-network"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Momentum, Adam, and what the optimiser is doing for you', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/optimisers-beyond-sgd (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Momentum, Adam, and what the optimiser is doing for you","courseTitle":"Machine Learning, Foundations","module":{"index":8,"slug":"neural-networks","title":"Neural networks, built from what you already know"},"answer":null,"keyIdea":"Momentum averages recent gradients so oscillations cancel and consistent directions accumulate; Adam adds a per-parameter step size from the average squared gradient, and its weight decay must be applied outside that scaling, which is what AdamW does.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-plain-sgd-struggles-with","runs":[{"text":"What plain SGD struggles with"}]},{"t":"p","runs":[{"text":"Gradient descent follows the slope where it currently stands. Two situations make that inefficient."}]},{"t":"p","runs":[{"text":"Ravines.","b":true},{"text":" A surface that is steep across and shallow along: the long narrow valley from module 2. The gradient points mostly across the valley, so the optimiser zigzags from wall to wall and makes slow progress along the floor."}]},{"t":"p","runs":[{"text":"Uneven scales.","b":true},{"text":" One parameter needs large steps and another needs small ones, and a single learning rate has to serve both."}]},{"t":"h2","id":"momentum","runs":[{"text":"Momentum"}]},{"t":"p","runs":[{"text":"Keep a running average of recent gradients and step in that direction."}]},{"t":"code","lang":"text","label":"Text","text":"v = beta * v + gradient          # beta typically 0.9\nw = w - learning_rate * v"},{"t":"p","runs":[{"text":"The zigzag components across the ravine alternate in sign and cancel in the average. The consistent component along the floor accumulates. The result is faster progress in the useful direction and damped oscillation, and it also carries the optimiser through small bumps and flat regions where the raw gradient is near zero."}]},{"t":"p","runs":[{"text":"With "},{"text":"beta = 0.9","code":true},{"text":", the effective step is roughly ten times the current gradient once the average has built up, which is why momentum with a given learning rate is considerably more aggressive than plain SGD with the same rate."}]},{"t":"p","runs":[{"text":"Nesterov momentum","b":true},{"text":" evaluates the gradient at the position the momentum is about to take you rather than where you are: a lookahead that damps overshoot slightly. A small, real improvement."}]},{"t":"h2","id":"adam","runs":[{"text":"Adam"}]},{"t":"p","runs":[{"text":"Adam adds a per-parameter learning rate on top of momentum. It keeps two running averages: one of the gradient, and one of the gradient "},{"text":"squared","i":true},{"text":". Then it divides the step by the square root of the second."}]},{"t":"code","lang":"text","label":"Text","text":"m = b1 * m + (1 - b1) * g          # average gradient\nv = b2 * v + (1 - b2) * g * g      # average squared gradient\nw = w - lr * m / (sqrt(v) + eps)"},{"t":"p","runs":[{"text":"The effect: a parameter that consistently receives large gradients has a large "},{"text":"v","code":true},{"text":", so its steps are scaled down; one receiving small gradients gets steps scaled up. Every parameter ends up with a step size suited to its own history, which removes most of the sensitivity to feature scaling and to how the layers happen to be sized."}]},{"t":"p","runs":[{"text":"Defaults "},{"text":"b1 = 0.9","code":true},{"text":", "},{"text":"b2 = 0.999","code":true},{"text":", "},{"text":"lr = 0.001","code":true},{"text":" work across an enormous range of problems, which is why Adam is what almost everyone uses by default."}]},{"t":"h2","id":"adam-s-costs","runs":[{"text":"Adam’s costs"}]},{"t":"p","runs":[{"text":"Memory.","b":true},{"text":" Two extra values per parameter. For a model with a billion parameters, the optimiser state is twice the size of the model, and it is a large part of why training needs so much more memory than inference."}]},{"t":"p","runs":[{"text":"Generalisation.","b":true},{"text":" On image classification specifically, well-tuned SGD with momentum often generalises slightly better than Adam. The gap is small and the tuning effort is not, and Adam remains the right default when you do not have time to tune."}]},{"t":"p","runs":[{"text":"Weight decay is broken in plain Adam.","b":true},{"text":" L2 regularisation added to the loss gets divided by the same "},{"text":"sqrt(v)","code":true},{"text":" term, so parameters with large gradients receive weaker decay, which is the opposite of intended. "},{"text":"AdamW","b":true},{"text":" applies the decay directly to the weights, outside the adaptive scaling, and it is what you should use."}]},{"t":"code","lang":"python","label":"Python","text":"torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)"},{"t":"p","runs":[{"text":"AdamW is the standard for training transformers and should be your default over Adam. The fix is one letter and it matters."}]},{"t":"h2","id":"schedules","runs":[{"text":"Schedules"}]},{"t":"p","runs":[{"text":"The learning rate should not stay constant, for the reasons in module 2."}]},{"t":"p","runs":[{"text":"Warmup","b":true},{"text":": increase from near zero over the first few hundred to few thousand steps. Essential for transformers, where early gradients on freshly initialised weights are unreliable and a large early step does lasting damage."}]},{"t":"p","runs":[{"text":"Cosine decay","b":true},{"text":": fall smoothly to near zero over training. The current standard."}]},{"t":"p","runs":[{"text":"One-cycle","b":true},{"text":": warm up to a high rate, then decay below the starting point. Trains surprisingly fast and is built into fastai and PyTorch."}]},{"t":"code","lang":"python","label":"Python","text":"sched = torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=1e-3,\n                                            total_steps=n_steps)"},{"t":"h2","id":"gradient-clipping","runs":[{"text":"Gradient clipping"}]},{"t":"p","runs":[{"text":"One more safety measure. If the gradient’s norm exceeds a threshold, scale it down."}]},{"t":"code","lang":"python","label":"Python","text":"torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)"},{"t":"p","runs":[{"text":"This prevents a single unusual batch from taking a catastrophic step and destroying weights that took hours to train. It is standard for recurrent networks and transformers and costs essentially nothing. If your loss occasionally spikes to "},{"text":"NaN","code":true},{"text":" after training normally for a while, this is usually the fix."}]},{"t":"h2","id":"what-to-actually-use","runs":[{"text":"What to actually use"}]},{"t":"p","runs":[{"text":"AdamW at 3e-4, cosine decay with warmup, gradient clipping at 1.0. That combination trains almost anything without drama. Reach for SGD with momentum only if you are chasing the last fraction of a point on a vision benchmark and have the compute to tune it."}]}],"check":{"question":[{"text":"A team uses Adam with weight_decay=0.01 and finds regularisation appears weaker than expected on the parameters that change most. Why does switching to AdamW help?"}],"options":[{"text":[{"text":"Adam’s momentum term cancels the decay over successive steps, while AdamW omits momentum entirely."}],"correct":false,"tempting":[{"text":"AdamW retains momentum; removing it would change the optimiser’s behaviour far more than the observed regularisation difference."}]},{"text":[{"text":"AdamW uses a larger effective decay coefficient by default, so the same value regularises more strongly."}],"correct":false,"tempting":[{"text":"The coefficient’s meaning differs between the two, but the fix is where the decay is applied rather than a change in its magnitude."}]},{"text":[{"text":"AdamW disables the adaptive per-parameter scaling, so every weight receives the same decay."}],"correct":false,"tempting":[{"text":"AdamW keeps the adaptive step sizes for the gradient term; only the decay is moved outside them."}]},{"text":[{"text":"In Adam the L2 term enters the loss and is then divided by the square root of the average squared gradient, so parameters with large gradients get weaker decay; AdamW applies the decay directly to the weights, outside that scaling."}],"correct":true,"tempting":null}],"order":[2,3,0,1],"why":null},"prev":{"slug":"initialisation-and-normalisation","title":"Starting well, and staying well-behaved","minutes":9},"next":{"slug":"regularising-a-network","title":"Keeping a large network from memorising","minutes":8}}}