Where the training data comes from

The corpora that are actually used

Almost every open model of the last few years is built on a small number of public web corpora, mixed with code, books, papers and synthetic text. Knowing the names is worth an afternoon because the model card usually names them, and if it does not, that absence is itself information.

Common Crawl is a non-profit that has been crawling the web since 2008 and publishes monthly snapshots. Each snapshot is a few billion pages; the archive as a whole runs to hundreds of terabytes of compressed HTML. It is the raw material under nearly everything, and it is raw: boilerplate, navigation menus, spam, duplicated pages, and text in every language mixed together.

C4 is Google’s cleaned English extract of a single Common Crawl snapshot, around 750 GB of text. It became the reference example of what filtering does, partly because researchers later audited it and found what the filters had removed, including a disproportionate amount of text written in African American English and text about LGBT topics, because a naive bad-word filter treats reclaimed language as profanity.

The Pile was EleutherAI’s 825 GB mixed corpus: code, PubMed, arXiv, Stack Exchange, and a books collection called Books3. Books3 turned out to be roughly 190,000 pirated books, and it was removed after takedown notices in 2023. Models trained before that still contain what it taught them.

Dolma is AI2’s 3-trillion-token corpus, released with its tooling. FineWeb is Hugging Face’s 15-trillion-token filtered Common Crawl derivative, notable because the team published ablations: they trained small models on each filtering variant to test whether a filter actually improved anything, rather than assuming it did. That methodology is the useful thing to take away, more than the corpus itself.

What filtered actually means

When a card says the data was filtered and deduplicated, it usually means some subset of these steps:

• Language identification, keeping documents a classifier thinks are in the target languages. This step is where low-resource languages disappear, because the classifier is least accurate exactly where the data is thinnest.
• Deduplication, exact and near-duplicate, at document and sometimes paragraph level. This matters more than it sounds: duplicated text is memorised much more readily, and dedup measurably reduces verbatim regurgitation.
• Quality classification, often a small model trained to recognise text that looks like a reference page or a textbook. This is a taste judgement wearing a statistical hat.
• Safety and toxicity filtering, with the collateral damage described above.
• Benchmark decontamination, removing documents that overlap with known test sets. Only some labs do it, only for the benchmarks they thought of, and usually by n-gram overlap, which misses paraphrases.

None of these steps is neutral, and none is disclosed in enough detail to be reproduced for most commercial releases.

Opt-out, and its honest limits

Several mechanisms now exist for saying do not train on this.

• robots.txt blocking specific crawler agents, including Common Crawl’s CCBot and various lab-specific crawlers.
• The emerging ai.txt and llms.txt conventions, which are conventions rather than enforcement.
• Spawning’s Do Not Train registry and the opt-out fields on some dataset hosts.

Each of these has the same limitation, and it is a mechanical one rather than a legal one. They are prospective only. Blocking a crawler today does nothing about the snapshots taken in the last fifteen years, which are already published, already mirrored, and already inside models that are already trained. There is no mechanism to remove a page from a crawl that has been distributed, and no mechanism to remove what a trained model learned from it short of retraining.

That is worth saying plainly rather than implying otherwise, because a lot of writing on this subject leaves people believing that adding a line to a file protects work that was published in 2019.

What you can do in practice

If you need to know what is in a model, you have three moves and they get progressively weaker.

1. Search the corpus, where the corpus is published. grep over Dolma or FineWeb is slow but conclusive for exact strings. Hugging Face’s dataset search covers many public sets.
2. Probe the model. Prompt it with the first half of a document you suspect and see whether it completes the second half verbatim. This detects memorisation, which is a subset of inclusion; a model can have trained on something without memorising it.
3. Test on fresh material. Write new items in the same style and compare performance. A large drop on fresh items is evidence of contamination, without ever seeing the corpus.

For your own datasets, do the thing the labs mostly do not: write a dataset card. Where each part came from, what licence it carries, what you removed, and the date. It takes an hour and it is the only version of provenance anyone downstream will ever be able to check.