Bigger kept working, and the argument about why

The numbers first

The first GPT, in 2018, had 117 million adjustable numbers. The second, in 2019, had 1.5 billion. The third, in 2020, had 175 billion, and was trained on around 300 billion tokens of text. Google’s largest of 2022 had 540 billion. Each step was roughly ten times the previous one in cost, and each was better, in ways that the earlier lesson on 2022 described.

What that lesson did not tell you is that the improvement was predictable, and that the prediction is the most important practical discovery of the period.

The scaling laws

In January 2020 a team at OpenAI trained a series of models of different sizes on different amounts of text and plotted the results. The error on the next-token task fell as a straight line on a log-log chart (a power law) in each of three quantities: the number of parameters, the amount of text, and the compute spent. The line held across seven orders of magnitude. If you knew how much a run would cost, you could say in advance roughly how good it would be.

That is why a lab can commit a hundred million dollars to a training run without a demonstration first. The curve says what the money buys. Nothing in the two winters had that property.

The correction

In 2022 a team at DeepMind found that the 2020 recipe had the proportions wrong. Everyone had been building models with too many parameters and feeding them too little text. For a fixed compute budget, the best results came from roughly twenty tokens of text per parameter. GPT-3 had been trained on under two. Their demonstration was a model of 70 billion parameters, trained on 1.4 trillion tokens, that beat a 280-billion-parameter model from the same lab at a quarter of the size.

The effect on the industry was immediate and is still visible. Models got smaller and read far more. Meta’s 8-billion-parameter model of 2024 was trained on fifteen trillion tokens (nearly two thousand per parameter, a hundred times the “optimal” ratio) because a small model that has read a great deal is cheap to run for every user, and the cost of serving a popular model soon exceeds the cost of training it. What is compute-optimal for training is not what is optimal for a product.

The bitter lesson

In 2019 Richard Sutton, one of the founders of reinforcement learning, wrote a short essay that the field argues about to this day. Its claim: across seventy years, methods that encoded human knowledge about a problem were beaten, every time, by general methods that used more compute. Chess, Go, speech, vision. The researchers who built in their own understanding got early results and were overtaken by the ones who let the machine learn from scale.

He called it bitter because it is unflattering. It is also a fair summary of the previous four lessons, and it is the belief on which the current spending is built.

The argument about emergence

Here is the unsettled part, and it is worth understanding rather than picking a side.

In 2022 a paper catalogued abilities that appeared to switch on suddenly at scale: a model of one size scored near zero at three-digit arithmetic, and a model ten times larger scored well, with nothing in between. This was called emergence and it was alarming, because it implied that new abilities could appear without warning.

In 2023 another paper showed that most of these jumps disappear when you change the scoring. If you mark an arithmetic answer right only when every digit is correct, small improvements in per-digit accuracy show as nothing until they suddenly cross the line. If you score partial credit, the improvement is a smooth curve, right along the scaling law. The ability did not switch on. The ruler did.

Both papers are right about something. Capability does improve smoothly in the quantity the model is trained on. But the world often scores pass-or-fail (a legal form is filed correctly or it is not) and on those tasks a smooth improvement underneath still looks, from outside, like a sudden arrival. The honest position is that surprises are more likely to be measurement artefacts than new phenomena, and that this does not make them less surprising to the person affected.

What the laws do not promise

Three limits, each mechanical.

The laws predict the error on next-token prediction. They do not predict whether a model can do the thing you care about, which is a different quantity that correlates imperfectly with it.

The chart is logarithmic. Each equal step along it costs ten times more than the last. A curve that keeps going is not a curve that stays affordable.

And the laws describe what happens with more of the same data. Whether the same data exists to be had is the subject of a later lesson, and since 2024 the labs have opened a second axis (spending more compute at answer time rather than at training time) partly because the first one was getting expensive.