{"course":{"slug":"maths-you-need","title":"The Maths You Actually Need","url":"https://addaly.com/learn/maths-you-need","level":"intermediate","track":"deep"},"lesson":{"slug":"derivatives-and-descent","title":"Slopes, and why training is walking downhill","url":"https://addaly.com/learn/maths-you-need/derivatives-and-descent","position":20,"of":76,"minutes":9,"answer":"A derivative says how the loss moves when you nudge one number; training just steps against it, over and over.","keyIdea":"A derivative says how the loss moves when you nudge one number; training just steps against it, over and over."},"body":{"format":"markdown","text":"## The only question a derivative answers\n\nIf I nudge this number a tiny bit, how much does that number move, and in which direction?\n\nThat is a derivative. Not a limit, not a rule about `x^n` becoming `nx^(n-1)`. A sensitivity.\n\nMake it concrete. You are training a model. The loss right now is 2.6400. You increase one weight by 0.001 and recompute. The loss becomes 2.6431. It went up by 0.0031 when the weight went up by 0.001.\n\n```\nslope = 0.0031 / 0.001 = 3.1\n```\n\nThe derivative of the loss with respect to that weight is 3.1. Read it in units: loss per unit of weight. Positive means increasing the weight increases the loss, so to reduce the loss you should decrease the weight.\n\nThat is the entire content of the idea. Everything else is machinery for computing it fast.\n\n## The gradient is just all of them at once\n\nA model has more than one weight. GPT-2 small has 124 million. Do the same nudge test for each one (hold everything else still, wiggle this one, see what the loss does) and you get 124 million slopes.\n\nPack them into a list. That list is the gradient. It is a vector with one slot per parameter, and slot 7 means \"how sensitive is the loss to parameter 7, right now, right here\".\n\nNobody computes these by nudging. Backpropagation gets all of them in roughly the cost of one forward pass, using the chain rule. That is an engineering triumph and it does not change what the numbers mean.\n\n## Why walk instead of solve\n\nIn school you found a minimum by setting the derivative to zero and solving. Why not do that here?\n\nBecause you cannot. The loss of a neural network is a function of hundreds of millions of parameters, wrapped in nonlinearities, averaged over data. There is no formula to set to zero and rearrange. No closed form exists, and nobody expects one.\n\nSo you do the only thing left. Stand where you are, look at which way is downhill, take a step, look again.\n\n```python\nw = 0.4200\ngrad = 3.1        # d(loss)/dw at this point\nlr = 3e-4         # learning rate: how big a step\n\nw = w - lr * grad # 0.41907\n```\n\nThat is gradient descent, complete. The minus sign is the whole idea: the gradient points uphill, so you move against it. Then you do it again. Training a model is that line, executed a few hundred thousand times, on every parameter at once.\n\n## The learning rate, and why it is fiddly\n\nA derivative is honest only very near where you measured it. It tells you the slope under your feet, not the shape of the valley.\n\nStep too small and training crawls; you burn a week of GPU time getting nowhere. Step too big and you leap past the bottom and land higher up the far slope, then leap back further, and the loss goes to `nan` in about forty steps. Typical values for transformers land near `3e-4` for pretraining and `1e-5` or lower for fine-tuning, and those numbers are folklore refined by thousands of failed runs, not derived from anything.\n\n## What the picture gets wrong\n\nEvery tutorial draws a smooth bowl with a ball rolling to the bottom. That drawing is doing you a small amount of harm.\n\nThe real loss surface has flat plateaus, narrow ravines where the gradient points across the ravine instead of along it, and vast numbers of points where the slope is near zero without being any kind of bottom. On top of that, you never see the true gradient. You see an estimate from one batch of examples, which is noisy: the next lesson but one is about exactly how noisy.\n\nAnd the goal is not the lowest point. A model that reached the true minimum of its training loss has memorised the training set and will be worse on anything new. You are walking downhill on a surface you cannot see, using a noisy compass, and you want to stop somewhere reasonable rather than at the bottom. It works far better than it has any right to, and no one fully agrees on why."},"check":{"question":"Partway through training, one weight has a gradient far larger than any other weight's. What does that actually tell you?","options":[{"text":"That weight is a long way from its best value and has far to travel.","correct":false,"tempting":"Reads the steepness of a slope as a distance to the bottom, when a slope only describes the spot you are standing on."},{"text":"The loss changes steeply if you nudge that weight right here, and nothing about how far away the minimum is.","correct":true},{"text":"That weight is the most important parameter in the network.","correct":false,"tempting":"Turns a local sensitivity into a claim about the whole model, when the same weight may be flat almost everywhere else."},{"text":"The learning rate is set too high for this model.","correct":false,"tempting":"Confuses the gradient, which is a property of the loss surface, with the step size, which you chose yourself."}]},"prev":{"slug":"rates-of-change-before-derivatives","title":"Slope, before anyone says the word derivative","url":"https://addaly.com/learn/maths-you-need/rates-of-change-before-derivatives"},"next":{"slug":"partial-derivatives-and-the-gradient","title":"Many knobs, one derivative each","url":"https://addaly.com/learn/maths-you-need/partial-derivatives-and-the-gradient"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Slopes, and why training is walking downhill', in The Maths You Actually Need. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/maths-you-need/derivatives-and-descent (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Slopes, and why training is walking downhill","courseTitle":"The Maths You Actually Need","module":{"index":3,"slug":"change-and-descent","title":"Change, slopes and the walk downhill"},"answer":null,"keyIdea":"A derivative says how the loss moves when you nudge one number; training just steps against it, over and over.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-only-question-a-derivative-answers","runs":[{"text":"The only question a derivative answers"}]},{"t":"p","runs":[{"text":"If I nudge this number a tiny bit, how much does that number move, and in which direction?"}]},{"t":"p","runs":[{"text":"That is a derivative. Not a limit, not a rule about "},{"text":"x^n","code":true},{"text":" becoming "},{"text":"nx^(n-1)","code":true},{"text":". A sensitivity."}]},{"t":"p","runs":[{"text":"Make it concrete. You are training a model. The loss right now is 2.6400. You increase one weight by 0.001 and recompute. The loss becomes 2.6431. It went up by 0.0031 when the weight went up by 0.001."}]},{"t":"code","lang":"","label":"Text","text":"slope = 0.0031 / 0.001 = 3.1"},{"t":"p","runs":[{"text":"The derivative of the loss with respect to that weight is 3.1. Read it in units: loss per unit of weight. Positive means increasing the weight increases the loss, so to reduce the loss you should decrease the weight."}]},{"t":"p","runs":[{"text":"That is the entire content of the idea. Everything else is machinery for computing it fast."}]},{"t":"h2","id":"the-gradient-is-just-all-of-them-at-once","runs":[{"text":"The gradient is just all of them at once"}]},{"t":"p","runs":[{"text":"A model has more than one weight. GPT-2 small has 124 million. Do the same nudge test for each one (hold everything else still, wiggle this one, see what the loss does) and you get 124 million slopes."}]},{"t":"p","runs":[{"text":"Pack them into a list. That list is the gradient. It is a vector with one slot per parameter, and slot 7 means “how sensitive is the loss to parameter 7, right now, right here”."}]},{"t":"p","runs":[{"text":"Nobody computes these by nudging. Backpropagation gets all of them in roughly the cost of one forward pass, using the chain rule. That is an engineering triumph and it does not change what the numbers mean."}]},{"t":"h2","id":"why-walk-instead-of-solve","runs":[{"text":"Why walk instead of solve"}]},{"t":"p","runs":[{"text":"In school you found a minimum by setting the derivative to zero and solving. Why not do that here?"}]},{"t":"p","runs":[{"text":"Because you cannot. The loss of a neural network is a function of hundreds of millions of parameters, wrapped in nonlinearities, averaged over data. There is no formula to set to zero and rearrange. No closed form exists, and nobody expects one."}]},{"t":"p","runs":[{"text":"So you do the only thing left. Stand where you are, look at which way is downhill, take a step, look again."}]},{"t":"code","lang":"python","label":"Python","text":"w = 0.4200\ngrad = 3.1        # d(loss)/dw at this point\nlr = 3e-4         # learning rate: how big a step\n\nw = w - lr * grad # 0.41907"},{"t":"p","runs":[{"text":"That is gradient descent, complete. The minus sign is the whole idea: the gradient points uphill, so you move against it. Then you do it again. Training a model is that line, executed a few hundred thousand times, on every parameter at once."}]},{"t":"h2","id":"the-learning-rate-and-why-it-is-fiddly","runs":[{"text":"The learning rate, and why it is fiddly"}]},{"t":"p","runs":[{"text":"A derivative is honest only very near where you measured it. It tells you the slope under your feet, not the shape of the valley."}]},{"t":"p","runs":[{"text":"Step too small and training crawls; you burn a week of GPU time getting nowhere. Step too big and you leap past the bottom and land higher up the far slope, then leap back further, and the loss goes to "},{"text":"nan","code":true},{"text":" in about forty steps. Typical values for transformers land near "},{"text":"3e-4","code":true},{"text":" for pretraining and "},{"text":"1e-5","code":true},{"text":" or lower for fine-tuning, and those numbers are folklore refined by thousands of failed runs, not derived from anything."}]},{"t":"h2","id":"what-the-picture-gets-wrong","runs":[{"text":"What the picture gets wrong"}]},{"t":"p","runs":[{"text":"Every tutorial draws a smooth bowl with a ball rolling to the bottom. That drawing is doing you a small amount of harm."}]},{"t":"p","runs":[{"text":"The real loss surface has flat plateaus, narrow ravines where the gradient points across the ravine instead of along it, and vast numbers of points where the slope is near zero without being any kind of bottom. On top of that, you never see the true gradient. You see an estimate from one batch of examples, which is noisy: the next lesson but one is about exactly how noisy."}]},{"t":"p","runs":[{"text":"And the goal is not the lowest point. A model that reached the true minimum of its training loss has memorised the training set and will be worse on anything new. You are walking downhill on a surface you cannot see, using a noisy compass, and you want to stop somewhere reasonable rather than at the bottom. It works far better than it has any right to, and no one fully agrees on why."}]}],"check":{"question":[{"text":"Partway through training, one weight has a gradient far larger than any other weight’s. What does that actually tell you?"}],"options":[{"text":[{"text":"That weight is a long way from its best value and has far to travel."}],"correct":false,"tempting":[{"text":"Reads the steepness of a slope as a distance to the bottom, when a slope only describes the spot you are standing on."}]},{"text":[{"text":"The loss changes steeply if you nudge that weight right here, and nothing about how far away the minimum is."}],"correct":true,"tempting":null},{"text":[{"text":"That weight is the most important parameter in the network."}],"correct":false,"tempting":[{"text":"Turns a local sensitivity into a claim about the whole model, when the same weight may be flat almost everywhere else."}]},{"text":[{"text":"The learning rate is set too high for this model."}],"correct":false,"tempting":[{"text":"Confuses the gradient, which is a property of the loss surface, with the step size, which you chose yourself."}]}],"order":[3,0,2,1],"why":null},"prev":{"slug":"rates-of-change-before-derivatives","title":"Slope, before anyone says the word derivative","minutes":8},"next":{"slug":"partial-derivatives-and-the-gradient","title":"Many knobs, one derivative each","minutes":8}}}