What is actually in the training data

The scale, first

A current model is trained on something in the range of 10 to 20 trillion tokens. At roughly four characters per token, 15 trillion tokens is around 60 terabytes of text. For comparison, English Wikipedia is about 20 gigabytes: three thousandths of one per cent of the corpus.

That ratio is worth holding on to. Wikipedia is heavily upsampled because it is clean and dense, but the bulk of what a model has read is ordinary web pages.

The layers of a corpus

Published recipes, and several labs have published them in detail, including the FineWeb, Dolma and RedPajama datasets, all freely downloadable, describe a similar stack:

• Web crawl, the majority by volume. Common Crawl is the usual starting point: petabytes of raw HTML collected over years.
• Code, from public repositories. Included heavily even for models not aimed at programming, because there is evidence it improves structured reasoning.
• Books and long-form text, valued for sustained argument over many pages, which web text rarely has. This is also the most legally contested category.
• Reference and academic sources: Wikipedia, arXiv, patents, government publications.
• Curated dialogue and forums, for conversational register.

Filtering is most of the work

Raw Common Crawl is unusable: navigation menus, cookie banners, spam, machine-translated slop and duplicated boilerplate. What separates a good corpus from a bad one is mostly the filtering pipeline.

Extraction. Getting readable text out of HTML while discarding chrome. Free tools such as trafilatura do this well and the choice measurably affects downstream quality.

Language identification. Usually a fastText classifier. It is imperfect on short documents and on code-switched text, which under-selects languages that mix scripts: a systematic disadvantage for exactly the multilingual users who were already paying more per token.

Quality classification. Some pipelines train a classifier to prefer text resembling a reference set of “good” documents. This works, and it embeds somebody’s definition of good. A filter trained to prefer Wikipedia-like prose downweights informal writing, dialect and much of the web where ordinary people actually write.

Deduplication. Both exact and near-duplicate removal, typically with MinHash. This is one of the highest-value steps: duplicated text increases verbatim memorisation and wastes compute, and published ablations show quality improving when it is done properly.

Safety and personal-data filtering. Removing illegal material, and attempting to strip identifiers. Attempting, because it is a heuristic exercise on 60 terabytes and nobody claims it is complete.

What follows for the model

Knowledge is uneven in a way that tracks the web. Topics well covered online are known well. A local regulation, a minority language, a recent event after the cutoff, or a subject discussed mostly offline: thin or absent. This is not a random gap; it correlates with which populations have historically published on the internet in a form crawlers reach.

Duplication produces memorisation. Sequences appearing many times can be reproduced verbatim, and studies have extracted training text from production models. Rare, and not zero, and it is one of the threads running through the copyright disputes in module eight.

Quality filters encode a judgement. Every filter is a decision about whose writing counts. That decision is rarely published in enough detail to audit, which is one reason fully open pipelines like OLMo and Dolma matter beyond their own model quality.

The cutoff, and why the model is confused about it

A model’s knowledge stops at its data cutoff. But the cutoff is soft: the crawl for a given date includes pages written earlier and discussions of events after the nominal date, and post-training data is usually more recent than pretraining data. So a model often has fragmentary knowledge of a period it “should not” know about, and is generally poor at reporting its own cutoff accurately.

Never ask a model what its cutoff is and treat the answer as fact. It is generating a plausible date, and providers publish the real one.

What you can do with this

Download a slice of an open corpus (FineWeb has small samples) and read a few hundred documents. It takes half an hour and it recalibrates expectations permanently. The text is more ordinary, more repetitive and more commercial than most people imagine, and knowing that makes the model’s strengths and its blind spots much easier to predict.