{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"how-weights-get-updated","title":"How a number inside the model gets changed","url":"https://addaly.com/learn/how-llms-work/how-weights-get-updated","position":42,"of":78,"minutes":9,"answer":"Training is a loop of measure the error, compute how each parameter contributed, nudge every parameter slightly, repeat, and the memory needed for the nudging is why training costs far more than inference.","keyIdea":"Training is a loop of measure the error, compute how each parameter contributed, nudge every parameter slightly, repeat, and the memory needed for the nudging is why training costs far more than inference."},"body":{"format":"markdown","text":"## The loop\n\nEverything in training is this, several hundred thousand times:\n\n1. Take a batch of text.\n2. Run it forward; compute the loss.\n3. Compute the **gradient**: for every one of the 8 billion parameters, how much would the loss change if this parameter changed slightly.\n4. Move every parameter a small step in the direction that reduces the loss.\n5. Repeat.\n\nStep 3 is backpropagation, and it is not magic. It is the chain rule from calculus applied systematically backwards through the network, reusing intermediate results so that computing all 8 billion partial derivatives costs about twice a forward pass rather than 8 billion times.\n\nThat efficiency is the reason deep learning is possible at all. Without it, the derivative for each parameter would need its own forward pass and training would be unthinkable.\n\n## The learning rate, and why it is the one you tune\n\nStep 4 needs a size. Too large and the update overshoots, the loss oscillates or explodes. Too small and training takes forever and can settle into a poor region.\n\nIn practice the learning rate follows a schedule: a short **warmup** from near zero over the first few thousand steps, then a peak, then a **decay** (usually cosine) down toward zero by the end. The warmup exists because early gradients are large and unreliable; the decay lets the model settle.\n\nTypical peak values are around 10⁻⁴ for pretraining a mid-size model and 10⁻⁵ or lower for fine-tuning. When a fine-tune produces nonsense, the learning rate being too high is the first suspect, every time.\n\n## Adam, and the memory it costs\n\nPlain gradient descent uses the gradient directly. Almost nothing does that any more. Adam and its variants keep two extra running averages per parameter (one of the gradient, one of its square) which lets each parameter get an effectively different step size.\n\nThe cost is memory, and this is where training budgets come from. Per parameter, in a standard mixed-precision setup:\n\n- 2 bytes for the weight in bf16\n- 4 bytes for a master copy in fp32\n- 4 bytes for the first moment\n- 4 bytes for the second moment\n- 2 bytes for the gradient\n\nAbout **16 bytes per parameter**, against 2 bytes for inference. An 8B model needs roughly 128 GB of optimiser and weight state before a single activation is stored, which is why fine-tuning a model that runs happily on one GPU may need eight.\n\n## Batches, and why they are huge\n\nGradients from a single example are noisy. Averaging over many gives a more reliable direction, and large batches keep the hardware busy. Pretraining batches of one to four million tokens are normal.\n\nYou rarely have the memory to hold that at once, so **gradient accumulation** is used: run several small batches, accumulate the gradients, then update once. Same result, less memory, more time.\n\n## Activations, the hidden cost\n\nThe backward pass needs the intermediate values from the forward pass, so they must be kept. For long sequences and large batches this can exceed the weights themselves. **Gradient checkpointing** trades compute for memory: discard most intermediates and recompute them during the backward pass, roughly 30 per cent more compute for a large memory saving. Almost every fine-tuning script offers a flag for it, and turning it on is usually the difference between fitting and not.\n\n## Why this matters even if you never train\n\nThree practical consequences.\n\n**Inference and training costs differ by an order of magnitude.** A model you can serve on one GPU may need a cluster to train. When someone proposes fine-tuning, the memory arithmetic above is the first check, and it is the reason the parameter-efficient methods in a later lesson exist.\n\n**Weights do not change during use.** The loop above runs during training and not during a conversation. Nothing you say updates anything.\n\n**A model cannot learn from its mistakes in production** without a deliberate retraining cycle: collect examples, curate them, run the loop, evaluate, deploy. That is a project, not a feature, and understanding why is the difference between a realistic roadmap and a wishful one."},"check":{"question":"A team can serve a 13-billion-parameter model comfortably on one 40 GB GPU but cannot fine-tune it on the same hardware. Why?","options":[{"text":"Fine-tuning requires the full training corpus in memory alongside the model.","correct":false,"tempting":"Data volume is an obvious scaling concern, and it is true that training touches far more data than inference."},{"text":"Training must additionally hold gradients, optimiser moments and a master copy of the weights: roughly 16 bytes per parameter against 2 for inference.","correct":true},{"text":"Backpropagation cannot be batched, so it needs a separate copy of the model per example.","correct":false,"tempting":"Backward passes are genuinely more complex, so imagining they resist batching is an understandable inference."},{"text":"Training runs at fp32 rather than bf16, which quadruples the weight memory.","correct":false,"tempting":"Higher precision is used for parts of training, so a precision-based explanation captures something real but misses the main cost."}]},"prev":{"slug":"the-loss-and-perplexity","title":"What the training objective actually measures","url":"https://addaly.com/learn/how-llms-work/the-loss-and-perplexity"},"next":{"slug":"scaling-laws","title":"Scaling laws, and the correction that changed everything","url":"https://addaly.com/learn/how-llms-work/scaling-laws"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'How a number inside the model gets changed', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/how-weights-get-updated (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"How a number inside the model gets changed","courseTitle":"How a Language Model Actually Works","module":{"index":5,"slug":"where-the-weights-come-from","title":"Where the weights come from"},"answer":null,"keyIdea":"Training is a loop of measure the error, compute how each parameter contributed, nudge every parameter slightly, repeat, and the memory needed for the nudging is why training costs far more than inference.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-loop","runs":[{"text":"The loop"}]},{"t":"p","runs":[{"text":"Everything in training is this, several hundred thousand times:"}]},{"t":"ol","start":1,"items":[[{"text":"Take a batch of text."}],[{"text":"Run it forward; compute the loss."}],[{"text":"Compute the "},{"text":"gradient","b":true},{"text":": for every one of the 8 billion parameters, how much would the loss change if this parameter changed slightly."}],[{"text":"Move every parameter a small step in the direction that reduces the loss."}],[{"text":"Repeat."}]]},{"t":"p","runs":[{"text":"Step 3 is backpropagation, and it is not magic. It is the chain rule from calculus applied systematically backwards through the network, reusing intermediate results so that computing all 8 billion partial derivatives costs about twice a forward pass rather than 8 billion times."}]},{"t":"p","runs":[{"text":"That efficiency is the reason deep learning is possible at all. Without it, the derivative for each parameter would need its own forward pass and training would be unthinkable."}]},{"t":"h2","id":"the-learning-rate-and-why-it-is-the-one-you-tune","runs":[{"text":"The learning rate, and why it is the one you tune"}]},{"t":"p","runs":[{"text":"Step 4 needs a size. Too large and the update overshoots, the loss oscillates or explodes. Too small and training takes forever and can settle into a poor region."}]},{"t":"p","runs":[{"text":"In practice the learning rate follows a schedule: a short "},{"text":"warmup","b":true},{"text":" from near zero over the first few thousand steps, then a peak, then a "},{"text":"decay","b":true},{"text":" (usually cosine) down toward zero by the end. The warmup exists because early gradients are large and unreliable; the decay lets the model settle."}]},{"t":"p","runs":[{"text":"Typical peak values are around 10⁻⁴ for pretraining a mid-size model and 10⁻⁵ or lower for fine-tuning. When a fine-tune produces nonsense, the learning rate being too high is the first suspect, every time."}]},{"t":"h2","id":"adam-and-the-memory-it-costs","runs":[{"text":"Adam, and the memory it costs"}]},{"t":"p","runs":[{"text":"Plain gradient descent uses the gradient directly. Almost nothing does that any more. Adam and its variants keep two extra running averages per parameter (one of the gradient, one of its square) which lets each parameter get an effectively different step size."}]},{"t":"p","runs":[{"text":"The cost is memory, and this is where training budgets come from. Per parameter, in a standard mixed-precision setup:"}]},{"t":"ul","items":[[{"text":"2 bytes for the weight in bf16"}],[{"text":"4 bytes for a master copy in fp32"}],[{"text":"4 bytes for the first moment"}],[{"text":"4 bytes for the second moment"}],[{"text":"2 bytes for the gradient"}]]},{"t":"p","runs":[{"text":"About "},{"text":"16 bytes per parameter","b":true},{"text":", against 2 bytes for inference. An 8B model needs roughly 128 GB of optimiser and weight state before a single activation is stored, which is why fine-tuning a model that runs happily on one GPU may need eight."}]},{"t":"h2","id":"batches-and-why-they-are-huge","runs":[{"text":"Batches, and why they are huge"}]},{"t":"p","runs":[{"text":"Gradients from a single example are noisy. Averaging over many gives a more reliable direction, and large batches keep the hardware busy. Pretraining batches of one to four million tokens are normal."}]},{"t":"p","runs":[{"text":"You rarely have the memory to hold that at once, so "},{"text":"gradient accumulation","b":true},{"text":" is used: run several small batches, accumulate the gradients, then update once. Same result, less memory, more time."}]},{"t":"h2","id":"activations-the-hidden-cost","runs":[{"text":"Activations, the hidden cost"}]},{"t":"p","runs":[{"text":"The backward pass needs the intermediate values from the forward pass, so they must be kept. For long sequences and large batches this can exceed the weights themselves. "},{"text":"Gradient checkpointing","b":true},{"text":" trades compute for memory: discard most intermediates and recompute them during the backward pass, roughly 30 per cent more compute for a large memory saving. Almost every fine-tuning script offers a flag for it, and turning it on is usually the difference between fitting and not."}]},{"t":"h2","id":"why-this-matters-even-if-you-never-train","runs":[{"text":"Why this matters even if you never train"}]},{"t":"p","runs":[{"text":"Three practical consequences."}]},{"t":"p","runs":[{"text":"Inference and training costs differ by an order of magnitude.","b":true},{"text":" A model you can serve on one GPU may need a cluster to train. When someone proposes fine-tuning, the memory arithmetic above is the first check, and it is the reason the parameter-efficient methods in a later lesson exist."}]},{"t":"p","runs":[{"text":"Weights do not change during use.","b":true},{"text":" The loop above runs during training and not during a conversation. Nothing you say updates anything."}]},{"t":"p","runs":[{"text":"A model cannot learn from its mistakes in production","b":true},{"text":" without a deliberate retraining cycle: collect examples, curate them, run the loop, evaluate, deploy. That is a project, not a feature, and understanding why is the difference between a realistic roadmap and a wishful one."}]}],"check":{"question":[{"text":"A team can serve a 13-billion-parameter model comfortably on one 40 GB GPU but cannot fine-tune it on the same hardware. Why?"}],"options":[{"text":[{"text":"Fine-tuning requires the full training corpus in memory alongside the model."}],"correct":false,"tempting":[{"text":"Data volume is an obvious scaling concern, and it is true that training touches far more data than inference."}]},{"text":[{"text":"Training must additionally hold gradients, optimiser moments and a master copy of the weights: roughly 16 bytes per parameter against 2 for inference."}],"correct":true,"tempting":null},{"text":[{"text":"Backpropagation cannot be batched, so it needs a separate copy of the model per example."}],"correct":false,"tempting":[{"text":"Backward passes are genuinely more complex, so imagining they resist batching is an understandable inference."}]},{"text":[{"text":"Training runs at fp32 rather than bf16, which quadruples the weight memory."}],"correct":false,"tempting":[{"text":"Higher precision is used for parts of training, so a precision-based explanation captures something real but misses the main cost."}]}],"order":[3,2,0,1],"why":null},"prev":{"slug":"the-loss-and-perplexity","title":"What the training objective actually measures","minutes":8},"next":{"slug":"scaling-laws","title":"Scaling laws, and the correction that changed everything","minutes":9}}}