Where the text came from, and whether it is running out

The pile

The chatbot module told you the model learned from “the pile”. This lesson is about what was in it, because the contents explain the voice, the gaps, and several lawsuits.

The base of nearly every large model is a crawl of the public web. The largest is run by a non-profit that has been saving copies of web pages since 2008, and it now holds petabytes across hundreds of billions of pages. It is free to anyone, and it is also, by language, about forty-five per cent English. Hindi is a fraction of one per cent, which is the mechanism behind the lesson on being weaker in your language.

On top of the crawl sit the sources a lab thinks are higher quality: the whole of Wikipedia, digitised books, academic papers, public code from GitHub, question-and-answer sites, forum threads, and transcripts of spoken video. Each has a story. A collection of nearly two hundred thousand books, assembled from a pirate library, was used in early open models and is now the central exhibit in several copyright cases. Forum and question-and-answer sites that were once crawled for free now sell access; one deal reported in 2024 was sixty million dollars a year.

The scale

GPT-3 was trained on about 300 billion tokens. Meta’s 2024 models were trained on fifteen trillion: fifty times more in four years. To place that: the whole of English Wikipedia is on the order of five billion tokens, which is about three hundredths of one per cent of what those models read. The encyclopaedia everyone imagines as the model’s main source is a rounding error in the mix. It matters more than its size suggests because labs give it extra weight, but it is not the pile.

The filter is the voice

Raw crawl is mostly unusable: navigation menus, boilerplate, spam, duplicated pages, machine-generated filler. Before training, a lab removes duplicates, strips the obvious junk, and then runs a classifier that keeps text resembling the sources it trusts (reference writing, published books, well-edited articles) and drops the rest.

The consequence is that the surviving text has a register. It reads like an encyclopaedia entry crossed with a helpful forum reply, because those are what the filter kept. When a model’s answer sounds like a Wikipedia paragraph with a friendly opening, you are hearing the filter, not a personality. The second training adjusts the manners; the register underneath was set by what was kept.

Running out

Here is the number that shapes the labs’ planning. A research group that tracks the industry estimated in 2024 that the total stock of public human-written text of usable quality is on the order of three hundred trillion tokens, and that at current growth in training-set size it will be fully used somewhere between 2026 and 2032. Text is not being written fast enough to keep up. The scaling laws say more text keeps helping; the world does not have it.

Four responses are under way, and the honest summary is that none is proven.

Licensing. Paying publishers, news agencies and forums for text that was not on the open web, or for the right to keep using what was. The deals are in the tens to hundreds of millions each.

Other kinds of data. Images, audio and video are far more abundant than text, and a model that reads them all is not limited by the text stock alone.

Synthetic text. Having a strong model write textbooks, exercises and dialogues for a smaller one to learn from. Several small models of 2023 and 2024 were trained heavily this way and performed above their size. Whether it works at the frontier (whether a model can teach a successor more than it knows) is open.

Reading the same text more than once. Which helps, with diminishing returns after a few passes.

The collapse worry

A 2024 paper in Nature trained models repeatedly on their own output and watched them degrade: each generation lost the rare cases and converged on the common ones, until the text became repetitive nonsense. The concern is that the web is now filling with model output, so future crawls will be partly a model’s own reflection.

The experiment was deliberately extreme, with each generation trained only on the last. Real training sets mix fresh human text with the rest, and subsequent work suggests the mix is stable if the human share is kept. But nobody yet knows the proportion of the 2025 web that is generated, and nobody can filter it reliably, because the detectors do not work. This is unsettled, and it is one of the few places where a mechanism you have already learned (the loss of the tails) points at a risk to the models themselves.