{"course":{"slug":"ai-actually-explained","title":"AI, Actually Explained","url":"https://addaly.com/learn/ai-actually-explained","level":"beginner","track":"start"},"lesson":{"slug":"one-architecture-ate-everything","title":"One architecture ate everything","url":"https://addaly.com/learn/ai-actually-explained/one-architecture-ate-everything","position":47,"of":92,"minutes":9,"answer":"The 2017 transformer replaced reading a sentence one word at a time with looking at every word at once, which let thousands of chips share the work, and the same design then took over images.","keyIdea":"The 2017 transformer replaced reading a sentence one word at a time with looking at every word at once, which let thousands of chips share the work, and the same design then took over images, speech and proteins."},"body":{"format":"markdown","text":"## Eight authors and a translation problem\n\nIn June 2017 eight researchers at Google published a paper with a title that was a joke about the field's habits: \"Attention Is All You Need\". It was written to translate English into German and French. It was not written to be the design under every chatbot in the world, and nothing in it suggests the authors expected that. All eight had left Google by 2023, most to found companies; one has since returned.\n\nThe previous lesson on the 2022 moment told you the transformer's importance was that it could be scaled. This lesson is about what it replaced, why the replacement could be scaled, and how one design for translating sentences ended up reading X-rays.\n\n## What it replaced\n\nUntil 2017 the standard way to handle a sentence was a recurrent network, which read it the way you do: one word at a time, carrying a summary forward. The best version dated from 1997 and had a mechanism for deciding what to keep in the summary and what to drop.\n\nIt had two problems, and they were the same problem. Reading one word at a time is sequential, so the ten thousand cores of a graphics chip mostly sat idle waiting for the previous word. And a summary carried across forty words has forgotten most of the first ten by the end, which is the local-fluency, global-incoherence failure you met in the statistical translators.\n\nThe transformer's move was to stop reading in order. Every word in the sentence is processed at the same time, and each one looks directly at every other to decide what matters: the attention you met in the chatbot module. Because nothing waits for anything, the whole sentence, and the whole batch of sentences, can be spread across every core of every chip at once. And because each word can look straight at any other, the fortieth word can attend to the first with no summary in between.\n\nOne thing had to be added back. If you process every word at once, you lose the order, and \"dog bites man\" is not \"man bites dog\". So the design stamps each word with a position before it goes in. That detail is why a model can know a word came third even though it never read the words in sequence.\n\n## The 2018 fork\n\nWithin eighteen months the field split the design in two and bet on both. One half kept the part that reads, and trained it to understand: given a sentence with a word hidden, predict the hidden word. Google's version, released in late 2018, quietly took over search ranking and the systems that classify text. The other half kept the part that writes, and trained it to continue: given the words so far, predict the next. That is the GPT line, and the lesson on the next-word game in this course is about it.\n\nThe writing half won the public's attention because it produces something you can read. Both halves are in use, and a great deal of the invisible AI in the second module runs on the reading half.\n\n## Pretrain once, adapt everywhere\n\nThe other change the design enabled was a habit. Because one large model trained on general text turned out to be a good starting point for almost any task with text in it, the field stopped training a model per task and started training one model, then adjusting it. The second training you met in the chatbot module, where an assistant is made from a text predictor, is this habit applied to conversation.\n\n## Then it ate the rest\n\nBy 2020 a group at Google had cut images into a grid of small squares, treated each square as if it were a word, and fed the grid to a transformer. It matched the best image networks. In 2022 a speech model trained on 680,000 hours of audio used the same design and transcribed in dozens of languages. The system that predicted protein structures well enough to be called solved in 2020 has attention at its centre. Code, music, video and protein all followed, for the same reason: they are all sequences, and the design did not care what the sequence was made of.\n\n> Figure: What the transformer replaced, and why it could be bought. Steps: 1. The whole sentence goes in at once (Nothing waits for the word before it, so the work spreads across every core of every chip); 2. Each word is stamped with its position (Order is lost when you stop reading in order, and 'dog bites man' is not 'man bites dog'); 3. Every word looks directly at every other (The fortieth word attends to the first with no summary in between); 4. Dozens of layers refine it (Early layers handle grammar-shaped relations, later ones meaning-shaped ones); 5. The same design takes images, speech and proteins (Cut a photograph into squares and treat each as a word. The design never cared what the sequence was made of). The old design read one word at a time, so the ten thousand cores of a graphics chip sat idle waiting, and a summary carried across forty words had forgotten the first ten. Processing everything at once fixed both, and made money convertible into capability, which is the whole story of the years since.\n\n## The cost the design carries\n\nAttention has one expensive property. Each word looks at every other, so the work grows with the square of the length. Double the text and the attention costs four times as much; ten times the text, a hundred times. That is the mechanism behind the context window's limits you met earlier, and behind the memory pressure in the previous lesson. A great deal of engineering since 2022 exists to work around it, and a family of alternative designs from 2023 onward processes sequences in linear time. None has displaced the transformer yet, partly because the entire industry's tooling and hardware are now shaped around it, and a design that everyone has optimised for is hard to beat even with a better idea."},"check":{"question":"Why did the transformer make it possible to turn more chips into a better model, when the previous designs had not?","options":[{"text":"It understood grammar better than a recurrent network, so it extracted more from the same text and needed fewer chips per improvement.","correct":false,"tempting":"Credits the design with understanding; its advantage was in how the work could be spread across hardware."},{"text":"It needed far less training data, so the shortage of text stopped being a limit.","correct":false,"tempting":"Reverses the effect; transformers are trained on more data than anything before them."},{"text":"It processed every word at once, so thousands of cores could share the work instead of waiting on the previous word.","correct":true},{"text":"It was smaller than a recurrent network and therefore fitted in a graphics card's memory.","correct":false,"tempting":"Picks the wrong resource; transformers are larger, and memory is the constraint they strain most."}]},"prev":{"slug":"the-chip-built-for-games","title":"The chip that was built for games","url":"https://addaly.com/learn/ai-actually-explained/the-chip-built-for-games"},"next":{"slug":"bigger-kept-working","title":"Bigger kept working, and the argument about why","url":"https://addaly.com/learn/ai-actually-explained/bigger-kept-working"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'One architecture ate everything', in AI, Actually Explained. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-actually-explained/one-architecture-ate-everything (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"One architecture ate everything","courseTitle":"AI, Actually Explained","module":{"index":5,"slug":"how-it-got-here","title":"How it got here, and what changed in 2022"},"answer":null,"keyIdea":"The 2017 transformer replaced reading a sentence one word at a time with looking at every word at once, which let thousands of chips share the work, and the same design then took over images, speech and proteins.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"The whole sentence goes in at once","detail":"Nothing waits for the word before it, so the work spreads across every core of every chip"},{"label":"Each word is stamped with its position","detail":"Order is lost when you stop reading in order, and ‘dog bites man’ is not ‘man bites dog’"},{"label":"Every word looks directly at every other","detail":"The fortieth word attends to the first with no summary in between"},{"label":"Dozens of layers refine it","detail":"Early layers handle grammar-shaped relations, later ones meaning-shaped ones"},{"label":"The same design takes images, speech and proteins","detail":"Cut a photograph into squares and treat each as a word. The design never cared what the sequence was made of"}],"title":"What the transformer replaced, and why it could be bought","caption":"The old design read one word at a time, so the ten thousand cores of a graphics chip sat idle waiting, and a summary carried across forty words had forgotten the first ten. Processing everything at once fixed both, and made money convertible into capability, which is the whole story of the years since."},"alt":"Figure: What the transformer replaced, and why it could be bought. Steps: 1. The whole sentence goes in at once (Nothing waits for the word before it, so the work spreads across every core of every chip); 2. Each word is stamped with its position (Order is lost when you stop reading in order, and ‘dog bites man’ is not ‘man bites dog’); 3. Every word looks directly at every other (The fortieth word attends to the first with no summary in between); 4. Dozens of layers refine it (Early layers handle grammar-shaped relations, later ones meaning-shaped ones); 5. The same design takes images, speech and proteins (Cut a photograph into squares and treat each as a word. The design never cared what the sequence was made of). The old design read one word at a time, so the ten thousand cores of a graphics chip sat idle waiting, and a summary carried across forty words had forgotten the first ten. Processing everything at once fixed both, and made money convertible into capability, which is the whole story of the years since."}},"blocks":[{"t":"h2","id":"eight-authors-and-a-translation-problem","runs":[{"text":"Eight authors and a translation problem"}]},{"t":"p","runs":[{"text":"In June 2017 eight researchers at Google published a paper with a title that was a joke about the field’s habits: “Attention Is All You Need”. It was written to translate English into German and French. It was not written to be the design under every chatbot in the world, and nothing in it suggests the authors expected that. All eight had left Google by 2023, most to found companies; one has since returned."}]},{"t":"p","runs":[{"text":"The previous lesson on the 2022 moment told you the transformer’s importance was that it could be scaled. This lesson is about what it replaced, why the replacement could be scaled, and how one design for translating sentences ended up reading X-rays."}]},{"t":"h2","id":"what-it-replaced","runs":[{"text":"What it replaced"}]},{"t":"p","runs":[{"text":"Until 2017 the standard way to handle a sentence was a recurrent network, which read it the way you do: one word at a time, carrying a summary forward. The best version dated from 1997 and had a mechanism for deciding what to keep in the summary and what to drop."}]},{"t":"p","runs":[{"text":"It had two problems, and they were the same problem. Reading one word at a time is sequential, so the ten thousand cores of a graphics chip mostly sat idle waiting for the previous word. And a summary carried across forty words has forgotten most of the first ten by the end, which is the local-fluency, global-incoherence failure you met in the statistical translators."}]},{"t":"p","runs":[{"text":"The transformer’s move was to stop reading in order. Every word in the sentence is processed at the same time, and each one looks directly at every other to decide what matters: the attention you met in the chatbot module. Because nothing waits for anything, the whole sentence, and the whole batch of sentences, can be spread across every core of every chip at once. And because each word can look straight at any other, the fortieth word can attend to the first with no summary in between."}]},{"t":"p","runs":[{"text":"One thing had to be added back. If you process every word at once, you lose the order, and “dog bites man” is not “man bites dog”. So the design stamps each word with a position before it goes in. That detail is why a model can know a word came third even though it never read the words in sequence."}]},{"t":"h2","id":"the-2018-fork","runs":[{"text":"The 2018 fork"}]},{"t":"p","runs":[{"text":"Within eighteen months the field split the design in two and bet on both. One half kept the part that reads, and trained it to understand: given a sentence with a word hidden, predict the hidden word. Google’s version, released in late 2018, quietly took over search ranking and the systems that classify text. The other half kept the part that writes, and trained it to continue: given the words so far, predict the next. That is the GPT line, and the lesson on the next-word game in this course is about it."}]},{"t":"p","runs":[{"text":"The writing half won the public’s attention because it produces something you can read. Both halves are in use, and a great deal of the invisible AI in the second module runs on the reading half."}]},{"t":"h2","id":"pretrain-once-adapt-everywhere","runs":[{"text":"Pretrain once, adapt everywhere"}]},{"t":"p","runs":[{"text":"The other change the design enabled was a habit. Because one large model trained on general text turned out to be a good starting point for almost any task with text in it, the field stopped training a model per task and started training one model, then adjusting it. The second training you met in the chatbot module, where an assistant is made from a text predictor, is this habit applied to conversation."}]},{"t":"h2","id":"then-it-ate-the-rest","runs":[{"text":"Then it ate the rest"}]},{"t":"p","runs":[{"text":"By 2020 a group at Google had cut images into a grid of small squares, treated each square as if it were a word, and fed the grid to a transformer. It matched the best image networks. In 2022 a speech model trained on 680,000 hours of audio used the same design and transcribed in dozens of languages. The system that predicted protein structures well enough to be called solved in 2020 has attention at its centre. Code, music, video and protein all followed, for the same reason: they are all sequences, and the design did not care what the sequence was made of."}]},{"t":"h2","id":"the-cost-the-design-carries","runs":[{"text":"The cost the design carries"}]},{"t":"p","runs":[{"text":"Attention has one expensive property. Each word looks at every other, so the work grows with the square of the length. Double the text and the attention costs four times as much; ten times the text, a hundred times. That is the mechanism behind the context window’s limits you met earlier, and behind the memory pressure in the previous lesson. A great deal of engineering since 2022 exists to work around it, and a family of alternative designs from 2023 onward processes sequences in linear time. None has displaced the transformer yet, partly because the entire industry’s tooling and hardware are now shaped around it, and a design that everyone has optimised for is hard to beat even with a better idea."}]}],"check":{"question":[{"text":"Why did the transformer make it possible to turn more chips into a better model, when the previous designs had not?"}],"options":[{"text":[{"text":"It understood grammar better than a recurrent network, so it extracted more from the same text and needed fewer chips per improvement."}],"correct":false,"tempting":[{"text":"Credits the design with understanding; its advantage was in how the work could be spread across hardware."}]},{"text":[{"text":"It needed far less training data, so the shortage of text stopped being a limit."}],"correct":false,"tempting":[{"text":"Reverses the effect; transformers are trained on more data than anything before them."}]},{"text":[{"text":"It processed every word at once, so thousands of cores could share the work instead of waiting on the previous word."}],"correct":true,"tempting":null},{"text":[{"text":"It was smaller than a recurrent network and therefore fitted in a graphics card’s memory."}],"correct":false,"tempting":[{"text":"Picks the wrong resource; transformers are larger, and memory is the constraint they strain most."}]}],"order":[1,3,0,2],"why":null},"prev":{"slug":"the-chip-built-for-games","title":"The chip that was built for games","minutes":9},"next":{"slug":"bigger-kept-working","title":"Bigger kept working, and the argument about why","minutes":10}}}