{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"gradient-descent","title":"Gradient descent, or walking downhill in fog","url":"https://addaly.com/learn/machine-learning-foundations/gradient-descent","position":12,"of":92,"minutes":8,"answer":"Gradient descent only knows the slope where it is standing; the step size decides whether that knowledge helps or destroys it.","keyIdea":"Gradient descent only knows the slope where it is standing; the step size decides whether that knowledge helps or destroys it."},"body":{"format":"markdown","text":"## The picture\n\nYou are on a hillside in thick fog. You want the bottom of the valley. You cannot see it. But you can feel which way the ground slopes under your feet, so you take a step in the downhill direction, feel again, and step again.\n\nThat is gradient descent, complete. The hill is the loss as a function of the parameters. Your position is the current parameter values. The slope is the gradient. The step size is the learning rate.\n\n## An actual walk\n\nOne parameter, and a loss with a known bottom:\n\n```\nL(w) = (w - 3)²\n```\n\nThe slope at any `w` is `2(w - 3)`. Start at `w = 0` with learning rate 0.1. The rule is: new `w` = old `w` − learning rate × slope.\n\n| Step | w | Slope | New w |\n|---|---|---|---|\n| 1 | 0.000 | −6.00 | 0.600 |\n| 2 | 0.600 | −4.80 | 1.080 |\n| 3 | 1.080 | −3.84 | 1.464 |\n| 4 | 1.464 | −3.07 | 1.771 |\n| … | | | |\n| 20 | 2.965 | −0.07 | 2.972 |\n\nIt closes in on 3, taking smaller steps as the slope flattens, because near the bottom there is less slope to push it. Nobody told it the answer was 3. It only ever felt the ground where it stood.\n\nNow set the learning rate to 1.1 and start again:\n\n| Step | w | Slope | New w |\n|---|---|---|---|\n| 1 | 0.00 | −6.00 | 6.60 |\n| 2 | 6.60 | 7.20 | −1.32 |\n| 3 | −1.32 | −8.64 | 8.18 |\n| 4 | 8.18 | 10.37 | −3.22 |\n\nEach step overshoots the bottom, lands further up the far side, and the next slope is steeper still. Within thirty steps the numbers exceed what a float can hold and your loss prints as `nan`. This is not an exotic failure. It is the single most common way training dies, and the fix is to lower the learning rate: usually by a factor of ten and then look again.\n\nToo large diverges. Too small crawls: at a learning rate of 0.0001 the walk above needs tens of thousands of steps. There is no formula for the right value. You try 0.1, 0.01, 0.001 on validation and watch the loss curve.\n\n## Real models have many parameters\n\nWith two parameters the loss is a surface and the gradient is a pair of numbers, one slope per parameter, each saying how much the loss changes when that one parameter moves. With ten million parameters it is a vector of ten million slopes, and the update rule is the same line of arithmetic applied to each. The mental picture does not need to scale; the arithmetic does, and it does.\n\n> Figure: Loss after each step, two learning rates, L(w) = (w − 3)². Line chart of Loss against Step: Learning rate 1.1: diverges (0, 9) (1, 12.96) (2, 18.66) (3, 26.87) (4, 38.7) (5, 55.7); Learning rate 0.1: converges (0, 9) (1, 5.76) (2, 3.69) (3, 2.36) (4, 1.51) (5, 0.97). From w = 0, a rate of 0.1 brings the loss from 9 to under 1 in five steps, each step smaller as the slope flattens. A rate of 1.1 overshoots the bottom every time and lands higher on the far side: 9, 13, 19, 27, 39, 56. Within thirty steps this prints as NaN.\n\n## How much data per step\n\nComputing the exact slope means running every training row through the model. With eight million rows that is one step per several minutes, which is unusable.\n\n**Mini-batch** gradient descent estimates the slope from a random sample (32, 128, 512 rows) and steps on that estimate. The estimate is noisy, so the path wanders rather than descending smoothly. That turns out to help: the noise shakes the walk out of shallow dips it would otherwise settle in, and it lets you take thousands of steps in the time one exact step would cost. Nearly all modern training is mini-batch.\n\n## Getting stuck\n\nThe standard worry is local minima: a small dip that is not the true bottom, where the slope is zero in every direction and the walk stops.\n\nFor the classic models this cannot happen. Linear regression, logistic regression and linear SVMs have convex losses: a single bowl, one bottom, and gradient descent finds it from anywhere.\n\nFor neural networks there are many minima, and for a long time this was assumed to be the central difficulty. It largely was not. In high dimensions the common flat spots are **saddle points**, where the surface rises in some directions and falls in others, and a walk can be slowed near one without being trapped. And when a large network does settle in a minimum, the minima it reaches tend to be about equally good, so which one you land in matters less than expected.\n\nWhat helps in practice is momentum (keep some of your previous direction, so you roll through flat regions instead of stalling) and adaptive methods like Adam, which give each parameter its own effective step size. Both are refinements of the same walk. Feel the slope, take a step, repeat."},"check":{"question":"You start training a network. Loss goes 4.2, then 91.4, then 3.1e6, then `nan` by step 30. What is the most likely cause?","options":[{"text":"The learning rate is too large, so each step overshoots and lands somewhere steeper than before.","correct":true},{"text":"The model is too small to fit the data.","correct":false,"tempting":"A terrible loss reads as insufficient capacity, but underfitting produces a loss that settles high and stays flat: it never explodes toward infinity."},{"text":"The optimiser is stuck in a local minimum.","correct":false,"tempting":"Local minima are the most repeated explanation for training failures in the whole field, which makes them the first thing people reach for, but being stuck makes a loss go flat, not infinite."},{"text":"There is not enough training data.","correct":false,"tempting":"Too little data explains a model that fails to generalise, which is a different symptom from a loss that diverges within thirty steps of starting."}]},"prev":{"slug":"the-closed-form-and-why-we-leave-it","title":"There is an exact answer, and we usually refuse it","url":"https://addaly.com/learn/machine-learning-foundations/the-closed-form-and-why-we-leave-it"},"next":{"slug":"batches-and-the-learning-rate","title":"Batches, epochs, and the two knobs that decide everything","url":"https://addaly.com/learn/machine-learning-foundations/batches-and-the-learning-rate"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Gradient descent, or walking downhill in fog', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/gradient-descent (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Gradient descent, or walking downhill in fog","courseTitle":"Machine Learning, Foundations","module":{"index":2,"slug":"linear-models-and-fitting","title":"Linear models and the machinery of fitting"},"answer":null,"keyIdea":"Gradient descent only knows the slope where it is standing; the step size decides whether that knowledge helps or destroys it.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"Loss after each step, two learning rates, L(w) = (w − 3)²","series":[{"label":"Learning rate 1.1: diverges","points":[[0,9],[1,12.96],[2,18.66],[3,26.87],[4,38.7],[5,55.7]]},{"label":"Learning rate 0.1: converges","points":[[0,9],[1,5.76],[2,3.69],[3,2.36],[4,1.51],[5,0.97]]}],"xLabel":"Step","yLabel":"Loss","caption":"From w = 0, a rate of 0.1 brings the loss from 9 to under 1 in five steps, each step smaller as the slope flattens. A rate of 1.1 overshoots the bottom every time and lands higher on the far side: 9, 13, 19, 27, 39, 56. Within thirty steps this prints as NaN."},"alt":"Figure: Loss after each step, two learning rates, L(w) = (w − 3)². Line chart of Loss against Step: Learning rate 1.1: diverges (0, 9) (1, 12.96) (2, 18.66) (3, 26.87) (4, 38.7) (5, 55.7); Learning rate 0.1: converges (0, 9) (1, 5.76) (2, 3.69) (3, 2.36) (4, 1.51) (5, 0.97). From w = 0, a rate of 0.1 brings the loss from 9 to under 1 in five steps, each step smaller as the slope flattens. A rate of 1.1 overshoots the bottom every time and lands higher on the far side: 9, 13, 19, 27, 39, 56. Within thirty steps this prints as NaN."}},"blocks":[{"t":"h2","id":"the-picture","runs":[{"text":"The picture"}]},{"t":"p","runs":[{"text":"You are on a hillside in thick fog. You want the bottom of the valley. You cannot see it. But you can feel which way the ground slopes under your feet, so you take a step in the downhill direction, feel again, and step again."}]},{"t":"p","runs":[{"text":"That is gradient descent, complete. The hill is the loss as a function of the parameters. Your position is the current parameter values. The slope is the gradient. The step size is the learning rate."}]},{"t":"h2","id":"an-actual-walk","runs":[{"text":"An actual walk"}]},{"t":"p","runs":[{"text":"One parameter, and a loss with a known bottom:"}]},{"t":"code","lang":"","label":"Text","text":"L(w) = (w - 3)²"},{"t":"p","runs":[{"text":"The slope at any "},{"text":"w","code":true},{"text":" is "},{"text":"2(w - 3)","code":true},{"text":". Start at "},{"text":"w = 0","code":true},{"text":" with learning rate 0.1. The rule is: new "},{"text":"w","code":true},{"text":" = old "},{"text":"w","code":true},{"text":" − learning rate × slope."}]},{"t":"table","head":[[{"text":"Step"}],[{"text":"w"}],[{"text":"Slope"}],[{"text":"New w"}]],"align":["left","left","left","left"],"rows":[[[{"text":"1"}],[{"text":"0.000"}],[{"text":"−6.00"}],[{"text":"0.600"}]],[[{"text":"2"}],[{"text":"0.600"}],[{"text":"−4.80"}],[{"text":"1.080"}]],[[{"text":"3"}],[{"text":"1.080"}],[{"text":"−3.84"}],[{"text":"1.464"}]],[[{"text":"4"}],[{"text":"1.464"}],[{"text":"−3.07"}],[{"text":"1.771"}]],[[{"text":"…"}],[],[],[]],[[{"text":"20"}],[{"text":"2.965"}],[{"text":"−0.07"}],[{"text":"2.972"}]]]},{"t":"p","runs":[{"text":"It closes in on 3, taking smaller steps as the slope flattens, because near the bottom there is less slope to push it. Nobody told it the answer was 3. It only ever felt the ground where it stood."}]},{"t":"p","runs":[{"text":"Now set the learning rate to 1.1 and start again:"}]},{"t":"table","head":[[{"text":"Step"}],[{"text":"w"}],[{"text":"Slope"}],[{"text":"New w"}]],"align":["left","left","left","left"],"rows":[[[{"text":"1"}],[{"text":"0.00"}],[{"text":"−6.00"}],[{"text":"6.60"}]],[[{"text":"2"}],[{"text":"6.60"}],[{"text":"7.20"}],[{"text":"−1.32"}]],[[{"text":"3"}],[{"text":"−1.32"}],[{"text":"−8.64"}],[{"text":"8.18"}]],[[{"text":"4"}],[{"text":"8.18"}],[{"text":"10.37"}],[{"text":"−3.22"}]]]},{"t":"p","runs":[{"text":"Each step overshoots the bottom, lands further up the far side, and the next slope is steeper still. Within thirty steps the numbers exceed what a float can hold and your loss prints as "},{"text":"nan","code":true},{"text":". This is not an exotic failure. It is the single most common way training dies, and the fix is to lower the learning rate: usually by a factor of ten and then look again."}]},{"t":"p","runs":[{"text":"Too large diverges. Too small crawls: at a learning rate of 0.0001 the walk above needs tens of thousands of steps. There is no formula for the right value. You try 0.1, 0.01, 0.001 on validation and watch the loss curve."}]},{"t":"h2","id":"real-models-have-many-parameters","runs":[{"text":"Real models have many parameters"}]},{"t":"p","runs":[{"text":"With two parameters the loss is a surface and the gradient is a pair of numbers, one slope per parameter, each saying how much the loss changes when that one parameter moves. With ten million parameters it is a vector of ten million slopes, and the update rule is the same line of arithmetic applied to each. The mental picture does not need to scale; the arithmetic does, and it does."}]},{"t":"h2","id":"how-much-data-per-step","runs":[{"text":"How much data per step"}]},{"t":"p","runs":[{"text":"Computing the exact slope means running every training row through the model. With eight million rows that is one step per several minutes, which is unusable."}]},{"t":"p","runs":[{"text":"Mini-batch","b":true},{"text":" gradient descent estimates the slope from a random sample (32, 128, 512 rows) and steps on that estimate. The estimate is noisy, so the path wanders rather than descending smoothly. That turns out to help: the noise shakes the walk out of shallow dips it would otherwise settle in, and it lets you take thousands of steps in the time one exact step would cost. Nearly all modern training is mini-batch."}]},{"t":"h2","id":"getting-stuck","runs":[{"text":"Getting stuck"}]},{"t":"p","runs":[{"text":"The standard worry is local minima: a small dip that is not the true bottom, where the slope is zero in every direction and the walk stops."}]},{"t":"p","runs":[{"text":"For the classic models this cannot happen. Linear regression, logistic regression and linear SVMs have convex losses: a single bowl, one bottom, and gradient descent finds it from anywhere."}]},{"t":"p","runs":[{"text":"For neural networks there are many minima, and for a long time this was assumed to be the central difficulty. It largely was not. In high dimensions the common flat spots are "},{"text":"saddle points","b":true},{"text":", where the surface rises in some directions and falls in others, and a walk can be slowed near one without being trapped. And when a large network does settle in a minimum, the minima it reaches tend to be about equally good, so which one you land in matters less than expected."}]},{"t":"p","runs":[{"text":"What helps in practice is momentum (keep some of your previous direction, so you roll through flat regions instead of stalling) and adaptive methods like Adam, which give each parameter its own effective step size. Both are refinements of the same walk. Feel the slope, take a step, repeat."}]}],"check":{"question":[{"text":"You start training a network. Loss goes 4.2, then 91.4, then 3.1e6, then "},{"text":"nan","code":true},{"text":" by step 30. What is the most likely cause?"}],"options":[{"text":[{"text":"The learning rate is too large, so each step overshoots and lands somewhere steeper than before."}],"correct":true,"tempting":null},{"text":[{"text":"The model is too small to fit the data."}],"correct":false,"tempting":[{"text":"A terrible loss reads as insufficient capacity, but underfitting produces a loss that settles high and stays flat: it never explodes toward infinity."}]},{"text":[{"text":"The optimiser is stuck in a local minimum."}],"correct":false,"tempting":[{"text":"Local minima are the most repeated explanation for training failures in the whole field, which makes them the first thing people reach for, but being stuck makes a loss go flat, not infinite."}]},{"text":[{"text":"There is not enough training data."}],"correct":false,"tempting":[{"text":"Too little data explains a model that fails to generalise, which is a different symptom from a loss that diverges within thirty steps of starting."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"the-closed-form-and-why-we-leave-it","title":"There is an exact answer, and we usually refuse it","minutes":8},"next":{"slug":"batches-and-the-learning-rate","title":"Batches, epochs, and the two knobs that decide everything","minutes":9}}}