Scaling laws, and the correction that changed everything

The surprising regularity

Train models of many sizes on many amounts of data and plot the loss. You do not get a mess. You get straight lines on a log-log plot: loss falls as a power law in model size, in data, and in compute.

The practical value is prediction. A lab can train a series of small models, fit the curve, and forecast the loss of a model a hundred times larger before spending the money. Very few things in machine learning are predictable in advance; this is one, and it is why enormous training runs are commissioned at all.

Two cautions before going further. The laws describe loss, not capability on any particular task, and the mapping from loss to usefulness is not a straight line. And they hold within the regime studied: extrapolating far beyond it is an assumption, not a result.

The correction

Early scaling work led practitioners to spend additional budget mostly on parameters. Models grew enormous and were trained on comparatively little data.

In 2022 the Chinchilla paper re-ran the experiments more carefully and found the balance was wrong. For a fixed compute budget, parameters and training tokens should scale roughly together: approximately 20 tokens per parameter, against ratios nearer 2 in the models that preceded it.

The demonstration was direct: a 70-billion-parameter model trained on 1.4 trillion tokens outperformed a 175-billion-parameter model trained on 300 billion. Less than half the size, better results, cheaper to serve. Almost everything since has been trained closer to this balance.

Then the balance shifted again, for economic reasons

Chinchilla answers “what is the best model for a fixed training budget”. That is the wrong question if you plan to serve the model to millions of people, because then inference dominates lifetime cost.

So labs began deliberately overtraining small models: pushing far past 20 tokens per parameter to squeeze more capability into a size that is cheap to run. Ratios of 100 or even 1,000 tokens per parameter appear in current small models: Llama 3’s 8B model saw 15 trillion tokens, nearly 2,000 per parameter.

The returns diminish, and they do not stop. Compute-optimal and deployment-optimal are different objectives, and nearly every model you actually use is optimised for the second.

The data wall

If tokens scale with parameters, and the supply of high-quality human text is finite, there is a ceiling. Estimates of the usable public text stock vary considerably, and several credible analyses put the exhaustion point for high-quality text somewhere in the second half of this decade. Treat the specific date as contested.

The responses under way, each with open questions:

• Synthetic data. Generate training text with models. It works for some domains (verifiable ones especially, where correctness can be checked) and carries a risk of compounding errors when models train on their own output. How far it goes is unresolved.
• Multiple epochs. Repeat the data. Published work suggests up to about four passes costs little; beyond that, returns fall off sharply.
• Other modalities. Video, audio and images add data of a different kind, though whether they improve text capability is not established.
• Better use of the same data. Curriculum ordering, harder filtering, more targeted mixtures. Currently a very active area, and the direction with the least hype attached.

What this means for you

Parameter count is a weak signal of quality. A well-trained 8B model from this year beats a poorly-trained 70B from three years ago on most tasks. Compare by evaluation, not by size.

Small models improved faster than large ones. Most of the gain came from training smaller models much longer and then distilling. That is why capable models now fit on a laptop, and it is the single most important trend for anyone without a budget.

Progress is not guaranteed to continue at the same rate. The gains of the last few years came from a combination of scale, better data, and post-training methods. Scale alone is running into cost and data limits. Reasonable people disagree about what happens next, and anyone claiming certainty in either direction is overreaching.