What was in the pile

The pile decides the model

A model is a compression of what it was trained on. So the honest way to predict what it will be good at is to ask what was in the pile.

For a large language model the pile is text on the order of ten to fifteen trillion tokens: a quantity nobody could read, roughly equivalent to tens of millions of books. It is assembled from a handful of recognisable sources:

• Web crawl. The bulk of it. Common Crawl, a free public archive of scraped web pages, is the starting point for most open efforts, filtered heavily.
• Wikipedia, in many languages, weighted far above its size because it is clean and factual.
• Books, from public-domain collections and, contentiously, from other sources.
• Code, largely from public repositories on GitHub. Training on code appears to improve reasoning on non-code tasks, which was not the plan.
• Forums and question sites (Reddit, Stack Exchange) which teach conversational structure and the shape of an answer.
• Scientific papers, from arXiv and PubMed.
• Increasingly, synthetic text generated by other models.

What is not in the pile

More useful than the list is its shadow.

Anything not written down. Craft knowledge that passes by demonstration (how a fabric should feel, how an engine sounds when the bearing is going) is thinly represented, because people who know it write it down rarely.

Anything behind a wall. Textbooks in copyright, most journalism after a paywall, court records that are not digitised, internal company documents, most of WhatsApp and every private conversation.

Anything in a language with a small web presence. This is the big one for most of the world. English is perhaps five per cent of humanity and a large majority of the training text. A language with a hundred million speakers and a thin internet is a language the model is weak in, and it will not say so.

Anything from before the internet that nobody scanned. Local history, regional legal practice, oral tradition.

Anything after the cut-off, which has its own lesson.

Filtering, and what filtering does

Raw crawl is largely unusable: boilerplate, navigation, spam, duplicates, machine-translated sludge. So it is filtered: deduplicated, scored by quality classifiers, and screened for certain categories of harmful content.

Deduplication matters more than it sounds. Training on near-identical copies of the same page makes memorisation likelier and generalisation worse, so labs spend serious effort on it.

But a quality classifier is itself a model with a view about what good text looks like, and that view was formed from examples somebody chose. Filtering is not neutral, and there is documented evidence that some filters disproportionately remove text written in non-standard dialects. Cleaning a corpus is an editorial act.

What is knowable, and what is now secret

This is worth being blunt about. Until around 2022, papers described training data in reasonable detail. Now the leading commercial labs do not disclose theirs, for competitive and legal reasons. When someone tells you what GPT-class models were trained on, they are inferring.

Open alternatives exist and are valuable if you want to see inside one:

• OLMo, from the Allen Institute for AI, ships with its full training corpus, Dolma, published and downloadable.
• The Pile and RedPajama are open, documented datasets you can inspect.
• Pythia is a set of models released with their data and intermediate checkpoints specifically so researchers can study what training does.

If you ever want to move from believing things about training data to checking them, those are the doors.

The one habit to take away

Before trusting a model on a subject, ask: was this written down, in quantity, in public, in this language, before the cut-off?

Four yeses and the model is probably solid. A no on any of them and you should expect confident output of unknown quality, because the model’s fluency is uniform, and its knowledge is not.

That asymmetry, between even fluency and uneven knowledge, is the single most dangerous property of these systems, and it comes directly from the pile.