One architecture ate everything

Eight authors and a translation problem

In June 2017 eight researchers at Google published a paper with a title that was a joke about the field’s habits: “Attention Is All You Need”. It was written to translate English into German and French. It was not written to be the design under every chatbot in the world, and nothing in it suggests the authors expected that. All eight had left Google by 2023, most to found companies; one has since returned.

The previous lesson on the 2022 moment told you the transformer’s importance was that it could be scaled. This lesson is about what it replaced, why the replacement could be scaled, and how one design for translating sentences ended up reading X-rays.

What it replaced

Until 2017 the standard way to handle a sentence was a recurrent network, which read it the way you do: one word at a time, carrying a summary forward. The best version dated from 1997 and had a mechanism for deciding what to keep in the summary and what to drop.

It had two problems, and they were the same problem. Reading one word at a time is sequential, so the ten thousand cores of a graphics chip mostly sat idle waiting for the previous word. And a summary carried across forty words has forgotten most of the first ten by the end, which is the local-fluency, global-incoherence failure you met in the statistical translators.

The transformer’s move was to stop reading in order. Every word in the sentence is processed at the same time, and each one looks directly at every other to decide what matters: the attention you met in the chatbot module. Because nothing waits for anything, the whole sentence, and the whole batch of sentences, can be spread across every core of every chip at once. And because each word can look straight at any other, the fortieth word can attend to the first with no summary in between.

One thing had to be added back. If you process every word at once, you lose the order, and “dog bites man” is not “man bites dog”. So the design stamps each word with a position before it goes in. That detail is why a model can know a word came third even though it never read the words in sequence.

The 2018 fork

Within eighteen months the field split the design in two and bet on both. One half kept the part that reads, and trained it to understand: given a sentence with a word hidden, predict the hidden word. Google’s version, released in late 2018, quietly took over search ranking and the systems that classify text. The other half kept the part that writes, and trained it to continue: given the words so far, predict the next. That is the GPT line, and the lesson on the next-word game in this course is about it.

The writing half won the public’s attention because it produces something you can read. Both halves are in use, and a great deal of the invisible AI in the second module runs on the reading half.

Pretrain once, adapt everywhere

The other change the design enabled was a habit. Because one large model trained on general text turned out to be a good starting point for almost any task with text in it, the field stopped training a model per task and started training one model, then adjusting it. The second training you met in the chatbot module, where an assistant is made from a text predictor, is this habit applied to conversation.

Then it ate the rest

By 2020 a group at Google had cut images into a grid of small squares, treated each square as if it were a word, and fed the grid to a transformer. It matched the best image networks. In 2022 a speech model trained on 680,000 hours of audio used the same design and transcribed in dozens of languages. The system that predicted protein structures well enough to be called solved in 2020 has attention at its centre. Code, music, video and protein all followed, for the same reason: they are all sequences, and the design did not care what the sequence was made of.

The cost the design carries

Attention has one expensive property. Each word looks at every other, so the work grows with the square of the length. Double the text and the attention costs four times as much; ten times the text, a hundred times. That is the mechanism behind the context window’s limits you met earlier, and behind the memory pressure in the previous lesson. A great deal of engineering since 2022 exists to work around it, and a family of alternative designs from 2023 onward processes sequences in linear time. None has displaced the transformer yet, partly because the entire industry’s tooling and hardware are now shaped around it, and a design that everyone has optimised for is hard to beat even with a better idea.