{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"scaling-laws","title":"Scaling laws, and the correction that changed everything","url":"https://addaly.com/learn/how-llms-work/scaling-laws","position":43,"of":78,"minutes":9,"answer":"Loss falls predictably as a power law in parameters, data and compute, and the 2022 correction showing most models were badly undertrained is why small, capable models exist.","keyIdea":"Loss falls predictably as a power law in parameters, data and compute, and the 2022 correction showing most models were badly undertrained is why small, capable models exist."},"body":{"format":"markdown","text":"## The surprising regularity\n\nTrain models of many sizes on many amounts of data and plot the loss. You do not get a mess. You get straight lines on a log-log plot: loss falls as a power law in model size, in data, and in compute.\n\nThe practical value is prediction. A lab can train a series of small models, fit the curve, and forecast the loss of a model a hundred times larger before spending the money. Very few things in machine learning are predictable in advance; this is one, and it is why enormous training runs are commissioned at all.\n\nTwo cautions before going further. The laws describe **loss**, not capability on any particular task, and the mapping from loss to usefulness is not a straight line. And they hold within the regime studied: extrapolating far beyond it is an assumption, not a result.\n\n## The correction\n\nEarly scaling work led practitioners to spend additional budget mostly on parameters. Models grew enormous and were trained on comparatively little data.\n\nIn 2022 the Chinchilla paper re-ran the experiments more carefully and found the balance was wrong. For a fixed compute budget, parameters and training tokens should scale **roughly together**: approximately 20 tokens per parameter, against ratios nearer 2 in the models that preceded it.\n\nThe demonstration was direct: a 70-billion-parameter model trained on 1.4 trillion tokens outperformed a 175-billion-parameter model trained on 300 billion. Less than half the size, better results, cheaper to serve. Almost everything since has been trained closer to this balance.\n\n## Then the balance shifted again, for economic reasons\n\nChinchilla answers \"what is the best model for a fixed *training* budget\". That is the wrong question if you plan to serve the model to millions of people, because then inference dominates lifetime cost.\n\nSo labs began deliberately overtraining small models: pushing far past 20 tokens per parameter to squeeze more capability into a size that is cheap to run. Ratios of 100 or even 1,000 tokens per parameter appear in current small models: Llama 3's 8B model saw 15 trillion tokens, nearly 2,000 per parameter.\n\n> Figure: The same eight-billion model, trained on more and more. Line chart of Cross-entropy loss, lower is better against Training tokens, trillions: 8B dense model (0.16, 2.06) (1, 1.9) (3, 1.8) (7, 1.73) (15, 1.68) (30, 1.65). The first mark is Chinchilla-optimal for this size, at twenty tokens per parameter. Everything to the right of it is money a lab spends because it will serve the model for years and serving is what costs. The returns diminish and do not stop. The shape is the finding; the exact values depend on the data mixture.\n\nThe returns diminish, and they do not stop. Compute-optimal and deployment-optimal are different objectives, and nearly every model you actually use is optimised for the second.\n\n## The data wall\n\nIf tokens scale with parameters, and the supply of high-quality human text is finite, there is a ceiling. Estimates of the usable public text stock vary considerably, and several credible analyses put the exhaustion point for high-quality text somewhere in the second half of this decade. Treat the specific date as contested.\n\nThe responses under way, each with open questions:\n\n- **Synthetic data.** Generate training text with models. It works for some domains (verifiable ones especially, where correctness can be checked) and carries a risk of compounding errors when models train on their own output. How far it goes is unresolved.\n- **Multiple epochs.** Repeat the data. Published work suggests up to about four passes costs little; beyond that, returns fall off sharply.\n- **Other modalities.** Video, audio and images add data of a different kind, though whether they improve text capability is not established.\n- **Better use of the same data.** Curriculum ordering, harder filtering, more targeted mixtures. Currently a very active area, and the direction with the least hype attached.\n\n## What this means for you\n\n**Parameter count is a weak signal of quality.** A well-trained 8B model from this year beats a poorly-trained 70B from three years ago on most tasks. Compare by evaluation, not by size.\n\n**Small models improved faster than large ones.** Most of the gain came from training smaller models much longer and then distilling. That is why capable models now fit on a laptop, and it is the single most important trend for anyone without a budget.\n\n**Progress is not guaranteed to continue at the same rate.** The gains of the last few years came from a combination of scale, better data, and post-training methods. Scale alone is running into cost and data limits. Reasonable people disagree about what happens next, and anyone claiming certainty in either direction is overreaching."},"check":{"question":"A 7-billion-parameter model released this year outperforms a 65-billion-parameter model from three years ago on most benchmarks. What best explains it?","options":[{"text":"The smaller model uses a more efficient architecture that extracts more from each parameter.","correct":false,"tempting":"Architectural improvements are real and are the usual headline, so crediting them is the natural explanation."},{"text":"The smaller model is quantised and better matched to the evaluation hardware.","correct":false,"tempting":"Quantisation and deployment details do affect measured results, which makes this sound like an experienced practitioner's caveat."},{"text":"Benchmarks have been contaminated by the newer model's training data.","correct":false,"tempting":"Contamination genuinely inflates recent results, so scepticism here is warranted and often correct in part."},{"text":"The newer model was trained on far more tokens per parameter, and the older one was substantially undertrained by current standards.","correct":true}]},"prev":{"slug":"how-weights-get-updated","title":"How a number inside the model gets changed","url":"https://addaly.com/learn/how-llms-work/how-weights-get-updated"},"next":{"slug":"what-training-costs","title":"What a training run actually costs","url":"https://addaly.com/learn/how-llms-work/what-training-costs"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Scaling laws, and the correction that changed everything', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/scaling-laws (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Scaling laws, and the correction that changed everything","courseTitle":"How a Language Model Actually Works","module":{"index":5,"slug":"where-the-weights-come-from","title":"Where the weights come from"},"answer":null,"keyIdea":"Loss falls predictably as a power law in parameters, data and compute, and the 2022 correction showing most models were badly undertrained is why small, capable models exist.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"The same eight-billion model, trained on more and more","series":[{"label":"8B dense model","points":[[0.16,2.06],[1,1.9],[3,1.8],[7,1.73],[15,1.68],[30,1.65]]}],"xLabel":"Training tokens, trillions","yLabel":"Cross-entropy loss, lower is better","caption":"The first mark is Chinchilla-optimal for this size, at twenty tokens per parameter. Everything to the right of it is money a lab spends because it will serve the model for years and serving is what costs. The returns diminish and do not stop. The shape is the finding; the exact values depend on the data mixture."},"alt":"Figure: The same eight-billion model, trained on more and more. Line chart of Cross-entropy loss, lower is better against Training tokens, trillions: 8B dense model (0.16, 2.06) (1, 1.9) (3, 1.8) (7, 1.73) (15, 1.68) (30, 1.65). The first mark is Chinchilla-optimal for this size, at twenty tokens per parameter. Everything to the right of it is money a lab spends because it will serve the model for years and serving is what costs. The returns diminish and do not stop. The shape is the finding; the exact values depend on the data mixture."}},"blocks":[{"t":"h2","id":"the-surprising-regularity","runs":[{"text":"The surprising regularity"}]},{"t":"p","runs":[{"text":"Train models of many sizes on many amounts of data and plot the loss. You do not get a mess. You get straight lines on a log-log plot: loss falls as a power law in model size, in data, and in compute."}]},{"t":"p","runs":[{"text":"The practical value is prediction. A lab can train a series of small models, fit the curve, and forecast the loss of a model a hundred times larger before spending the money. Very few things in machine learning are predictable in advance; this is one, and it is why enormous training runs are commissioned at all."}]},{"t":"p","runs":[{"text":"Two cautions before going further. The laws describe "},{"text":"loss","b":true},{"text":", not capability on any particular task, and the mapping from loss to usefulness is not a straight line. And they hold within the regime studied: extrapolating far beyond it is an assumption, not a result."}]},{"t":"h2","id":"the-correction","runs":[{"text":"The correction"}]},{"t":"p","runs":[{"text":"Early scaling work led practitioners to spend additional budget mostly on parameters. Models grew enormous and were trained on comparatively little data."}]},{"t":"p","runs":[{"text":"In 2022 the Chinchilla paper re-ran the experiments more carefully and found the balance was wrong. For a fixed compute budget, parameters and training tokens should scale "},{"text":"roughly together","b":true},{"text":": approximately 20 tokens per parameter, against ratios nearer 2 in the models that preceded it."}]},{"t":"p","runs":[{"text":"The demonstration was direct: a 70-billion-parameter model trained on 1.4 trillion tokens outperformed a 175-billion-parameter model trained on 300 billion. Less than half the size, better results, cheaper to serve. Almost everything since has been trained closer to this balance."}]},{"t":"h2","id":"then-the-balance-shifted-again-for-economic-reasons","runs":[{"text":"Then the balance shifted again, for economic reasons"}]},{"t":"p","runs":[{"text":"Chinchilla answers “what is the best model for a fixed "},{"text":"training","i":true},{"text":" budget”. That is the wrong question if you plan to serve the model to millions of people, because then inference dominates lifetime cost."}]},{"t":"p","runs":[{"text":"So labs began deliberately overtraining small models: pushing far past 20 tokens per parameter to squeeze more capability into a size that is cheap to run. Ratios of 100 or even 1,000 tokens per parameter appear in current small models: Llama 3’s 8B model saw 15 trillion tokens, nearly 2,000 per parameter."}]},{"t":"p","runs":[{"text":"The returns diminish, and they do not stop. Compute-optimal and deployment-optimal are different objectives, and nearly every model you actually use is optimised for the second."}]},{"t":"h2","id":"the-data-wall","runs":[{"text":"The data wall"}]},{"t":"p","runs":[{"text":"If tokens scale with parameters, and the supply of high-quality human text is finite, there is a ceiling. Estimates of the usable public text stock vary considerably, and several credible analyses put the exhaustion point for high-quality text somewhere in the second half of this decade. Treat the specific date as contested."}]},{"t":"p","runs":[{"text":"The responses under way, each with open questions:"}]},{"t":"ul","items":[[{"text":"Synthetic data.","b":true},{"text":" Generate training text with models. It works for some domains (verifiable ones especially, where correctness can be checked) and carries a risk of compounding errors when models train on their own output. How far it goes is unresolved."}],[{"text":"Multiple epochs.","b":true},{"text":" Repeat the data. Published work suggests up to about four passes costs little; beyond that, returns fall off sharply."}],[{"text":"Other modalities.","b":true},{"text":" Video, audio and images add data of a different kind, though whether they improve text capability is not established."}],[{"text":"Better use of the same data.","b":true},{"text":" Curriculum ordering, harder filtering, more targeted mixtures. Currently a very active area, and the direction with the least hype attached."}]]},{"t":"h2","id":"what-this-means-for-you","runs":[{"text":"What this means for you"}]},{"t":"p","runs":[{"text":"Parameter count is a weak signal of quality.","b":true},{"text":" A well-trained 8B model from this year beats a poorly-trained 70B from three years ago on most tasks. Compare by evaluation, not by size."}]},{"t":"p","runs":[{"text":"Small models improved faster than large ones.","b":true},{"text":" Most of the gain came from training smaller models much longer and then distilling. That is why capable models now fit on a laptop, and it is the single most important trend for anyone without a budget."}]},{"t":"p","runs":[{"text":"Progress is not guaranteed to continue at the same rate.","b":true},{"text":" The gains of the last few years came from a combination of scale, better data, and post-training methods. Scale alone is running into cost and data limits. Reasonable people disagree about what happens next, and anyone claiming certainty in either direction is overreaching."}]}],"check":{"question":[{"text":"A 7-billion-parameter model released this year outperforms a 65-billion-parameter model from three years ago on most benchmarks. What best explains it?"}],"options":[{"text":[{"text":"The smaller model uses a more efficient architecture that extracts more from each parameter."}],"correct":false,"tempting":[{"text":"Architectural improvements are real and are the usual headline, so crediting them is the natural explanation."}]},{"text":[{"text":"The smaller model is quantised and better matched to the evaluation hardware."}],"correct":false,"tempting":[{"text":"Quantisation and deployment details do affect measured results, which makes this sound like an experienced practitioner’s caveat."}]},{"text":[{"text":"Benchmarks have been contaminated by the newer model’s training data."}],"correct":false,"tempting":[{"text":"Contamination genuinely inflates recent results, so scepticism here is warranted and often correct in part."}]},{"text":[{"text":"The newer model was trained on far more tokens per parameter, and the older one was substantially undertrained by current standards."}],"correct":true,"tempting":null}],"order":[2,3,1,0],"why":null},"prev":{"slug":"how-weights-get-updated","title":"How a number inside the model gets changed","minutes":9},"next":{"slug":"what-training-costs","title":"What a training run actually costs","minutes":8}}}