{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"what-is-in-the-training-data","title":"What is actually in the training data","url":"https://addaly.com/learn/how-llms-work/what-is-in-the-training-data","position":40,"of":78,"minutes":9,"answer":"Pretraining corpora are mostly filtered web text, and the filtering decisions (what counts as quality, what gets deduplicated, what gets removed) shape the model as much as the architecture does.","keyIdea":"Pretraining corpora are mostly filtered web text, and the filtering decisions (what counts as quality, what gets deduplicated, what gets removed) shape the model as much as the architecture does."},"body":{"format":"markdown","text":"## The scale, first\n\nA current model is trained on something in the range of 10 to 20 trillion tokens. At roughly four characters per token, 15 trillion tokens is around 60 terabytes of text. For comparison, English Wikipedia is about 20 gigabytes: three thousandths of one per cent of the corpus.\n\nThat ratio is worth holding on to. Wikipedia is heavily upsampled because it is clean and dense, but the bulk of what a model has read is ordinary web pages.\n\n## The layers of a corpus\n\nPublished recipes, and several labs have published them in detail, including the FineWeb, Dolma and RedPajama datasets, all freely downloadable, describe a similar stack:\n\n- **Web crawl**, the majority by volume. Common Crawl is the usual starting point: petabytes of raw HTML collected over years.\n- **Code**, from public repositories. Included heavily even for models not aimed at programming, because there is evidence it improves structured reasoning.\n- **Books and long-form text**, valued for sustained argument over many pages, which web text rarely has. This is also the most legally contested category.\n- **Reference and academic sources**: Wikipedia, arXiv, patents, government publications.\n- **Curated dialogue and forums**, for conversational register.\n\n## Filtering is most of the work\n\nRaw Common Crawl is unusable: navigation menus, cookie banners, spam, machine-translated slop and duplicated boilerplate. What separates a good corpus from a bad one is mostly the filtering pipeline.\n\n**Extraction.** Getting readable text out of HTML while discarding chrome. Free tools such as `trafilatura` do this well and the choice measurably affects downstream quality.\n\n**Language identification.** Usually a fastText classifier. It is imperfect on short documents and on code-switched text, which under-selects languages that mix scripts: a systematic disadvantage for exactly the multilingual users who were already paying more per token.\n\n**Quality classification.** Some pipelines train a classifier to prefer text resembling a reference set of \"good\" documents. This works, and it embeds somebody's definition of good. A filter trained to prefer Wikipedia-like prose downweights informal writing, dialect and much of the web where ordinary people actually write.\n\n**Deduplication.** Both exact and near-duplicate removal, typically with MinHash. This is one of the highest-value steps: duplicated text increases verbatim memorisation and wastes compute, and published ablations show quality improving when it is done properly.\n\n**Safety and personal-data filtering.** Removing illegal material, and attempting to strip identifiers. Attempting, because it is a heuristic exercise on 60 terabytes and nobody claims it is complete.\n\n## What follows for the model\n\n**Knowledge is uneven in a way that tracks the web.** Topics well covered online are known well. A local regulation, a minority language, a recent event after the cutoff, or a subject discussed mostly offline: thin or absent. This is not a random gap; it correlates with which populations have historically published on the internet in a form crawlers reach.\n\n**Duplication produces memorisation.** Sequences appearing many times can be reproduced verbatim, and studies have extracted training text from production models. Rare, and not zero, and it is one of the threads running through the copyright disputes in module eight.\n\n**Quality filters encode a judgement.** Every filter is a decision about whose writing counts. That decision is rarely published in enough detail to audit, which is one reason fully open pipelines like OLMo and Dolma matter beyond their own model quality.\n\n## The cutoff, and why the model is confused about it\n\nA model's knowledge stops at its data cutoff. But the cutoff is soft: the crawl for a given date includes pages written earlier and discussions of events after the nominal date, and post-training data is usually more recent than pretraining data. So a model often has fragmentary knowledge of a period it \"should not\" know about, and is generally poor at reporting its own cutoff accurately.\n\nNever ask a model what its cutoff is and treat the answer as fact. It is generating a plausible date, and providers publish the real one.\n\n## What you can do with this\n\nDownload a slice of an open corpus (FineWeb has small samples) and read a few hundred documents. It takes half an hour and it recalibrates expectations permanently. The text is more ordinary, more repetitive and more commercial than most people imagine, and knowing that makes the model's strengths and its blind spots much easier to predict."},"check":{"question":"A model gives excellent answers about American tenancy law and poor, confidently wrong answers about tenancy rules in a mid-sized Indian city. What is the mechanism?","options":[{"text":"Coverage in the corpus follows what was published on the crawlable web, and the model produces a fluent continuation regardless of how thin its evidence was.","correct":true},{"text":"The model's safety training makes it cautious about non-Western legal topics.","correct":false,"tempting":"Safety tuning does affect legal and medical answers, so attributing a quality gap to caution is a reasonable read."},{"text":"Indian legal text is tokenised inefficiently, reducing the effective information the model received.","correct":false,"tempting":"Tokenisation costs are real for Indian-language text, which makes this technically informed but misapplied to English-language sources."},{"text":"The training corpus excluded non-English jurisdictions during language filtering.","correct":false,"tempting":"Language filtering does distort corpora, so a filtering explanation sounds well grounded even where the sources are in English."}]},"prev":{"slug":"pretraining-finetuning-rlhf","title":"Pretraining, fine-tuning, and preference training","url":"https://addaly.com/learn/how-llms-work/pretraining-finetuning-rlhf"},"next":{"slug":"the-loss-and-perplexity","title":"What the training objective actually measures","url":"https://addaly.com/learn/how-llms-work/the-loss-and-perplexity"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What is actually in the training data', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/what-is-in-the-training-data (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What is actually in the training data","courseTitle":"How a Language Model Actually Works","module":{"index":5,"slug":"where-the-weights-come-from","title":"Where the weights come from"},"answer":null,"keyIdea":"Pretraining corpora are mostly filtered web text, and the filtering decisions (what counts as quality, what gets deduplicated, what gets removed) shape the model as much as the architecture does.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-scale-first","runs":[{"text":"The scale, first"}]},{"t":"p","runs":[{"text":"A current model is trained on something in the range of 10 to 20 trillion tokens. At roughly four characters per token, 15 trillion tokens is around 60 terabytes of text. For comparison, English Wikipedia is about 20 gigabytes: three thousandths of one per cent of the corpus."}]},{"t":"p","runs":[{"text":"That ratio is worth holding on to. Wikipedia is heavily upsampled because it is clean and dense, but the bulk of what a model has read is ordinary web pages."}]},{"t":"h2","id":"the-layers-of-a-corpus","runs":[{"text":"The layers of a corpus"}]},{"t":"p","runs":[{"text":"Published recipes, and several labs have published them in detail, including the FineWeb, Dolma and RedPajama datasets, all freely downloadable, describe a similar stack:"}]},{"t":"ul","items":[[{"text":"Web crawl","b":true},{"text":", the majority by volume. Common Crawl is the usual starting point: petabytes of raw HTML collected over years."}],[{"text":"Code","b":true},{"text":", from public repositories. Included heavily even for models not aimed at programming, because there is evidence it improves structured reasoning."}],[{"text":"Books and long-form text","b":true},{"text":", valued for sustained argument over many pages, which web text rarely has. This is also the most legally contested category."}],[{"text":"Reference and academic sources","b":true},{"text":": Wikipedia, arXiv, patents, government publications."}],[{"text":"Curated dialogue and forums","b":true},{"text":", for conversational register."}]]},{"t":"h2","id":"filtering-is-most-of-the-work","runs":[{"text":"Filtering is most of the work"}]},{"t":"p","runs":[{"text":"Raw Common Crawl is unusable: navigation menus, cookie banners, spam, machine-translated slop and duplicated boilerplate. What separates a good corpus from a bad one is mostly the filtering pipeline."}]},{"t":"p","runs":[{"text":"Extraction.","b":true},{"text":" Getting readable text out of HTML while discarding chrome. Free tools such as "},{"text":"trafilatura","code":true},{"text":" do this well and the choice measurably affects downstream quality."}]},{"t":"p","runs":[{"text":"Language identification.","b":true},{"text":" Usually a fastText classifier. It is imperfect on short documents and on code-switched text, which under-selects languages that mix scripts: a systematic disadvantage for exactly the multilingual users who were already paying more per token."}]},{"t":"p","runs":[{"text":"Quality classification.","b":true},{"text":" Some pipelines train a classifier to prefer text resembling a reference set of “good” documents. This works, and it embeds somebody’s definition of good. A filter trained to prefer Wikipedia-like prose downweights informal writing, dialect and much of the web where ordinary people actually write."}]},{"t":"p","runs":[{"text":"Deduplication.","b":true},{"text":" Both exact and near-duplicate removal, typically with MinHash. This is one of the highest-value steps: duplicated text increases verbatim memorisation and wastes compute, and published ablations show quality improving when it is done properly."}]},{"t":"p","runs":[{"text":"Safety and personal-data filtering.","b":true},{"text":" Removing illegal material, and attempting to strip identifiers. Attempting, because it is a heuristic exercise on 60 terabytes and nobody claims it is complete."}]},{"t":"h2","id":"what-follows-for-the-model","runs":[{"text":"What follows for the model"}]},{"t":"p","runs":[{"text":"Knowledge is uneven in a way that tracks the web.","b":true},{"text":" Topics well covered online are known well. A local regulation, a minority language, a recent event after the cutoff, or a subject discussed mostly offline: thin or absent. This is not a random gap; it correlates with which populations have historically published on the internet in a form crawlers reach."}]},{"t":"p","runs":[{"text":"Duplication produces memorisation.","b":true},{"text":" Sequences appearing many times can be reproduced verbatim, and studies have extracted training text from production models. Rare, and not zero, and it is one of the threads running through the copyright disputes in module eight."}]},{"t":"p","runs":[{"text":"Quality filters encode a judgement.","b":true},{"text":" Every filter is a decision about whose writing counts. That decision is rarely published in enough detail to audit, which is one reason fully open pipelines like OLMo and Dolma matter beyond their own model quality."}]},{"t":"h2","id":"the-cutoff-and-why-the-model-is-confused-about-it","runs":[{"text":"The cutoff, and why the model is confused about it"}]},{"t":"p","runs":[{"text":"A model’s knowledge stops at its data cutoff. But the cutoff is soft: the crawl for a given date includes pages written earlier and discussions of events after the nominal date, and post-training data is usually more recent than pretraining data. So a model often has fragmentary knowledge of a period it “should not” know about, and is generally poor at reporting its own cutoff accurately."}]},{"t":"p","runs":[{"text":"Never ask a model what its cutoff is and treat the answer as fact. It is generating a plausible date, and providers publish the real one."}]},{"t":"h2","id":"what-you-can-do-with-this","runs":[{"text":"What you can do with this"}]},{"t":"p","runs":[{"text":"Download a slice of an open corpus (FineWeb has small samples) and read a few hundred documents. It takes half an hour and it recalibrates expectations permanently. The text is more ordinary, more repetitive and more commercial than most people imagine, and knowing that makes the model’s strengths and its blind spots much easier to predict."}]}],"check":{"question":[{"text":"A model gives excellent answers about American tenancy law and poor, confidently wrong answers about tenancy rules in a mid-sized Indian city. What is the mechanism?"}],"options":[{"text":[{"text":"Coverage in the corpus follows what was published on the crawlable web, and the model produces a fluent continuation regardless of how thin its evidence was."}],"correct":true,"tempting":null},{"text":[{"text":"The model’s safety training makes it cautious about non-Western legal topics."}],"correct":false,"tempting":[{"text":"Safety tuning does affect legal and medical answers, so attributing a quality gap to caution is a reasonable read."}]},{"text":[{"text":"Indian legal text is tokenised inefficiently, reducing the effective information the model received."}],"correct":false,"tempting":[{"text":"Tokenisation costs are real for Indian-language text, which makes this technically informed but misapplied to English-language sources."}]},{"text":[{"text":"The training corpus excluded non-English jurisdictions during language filtering."}],"correct":false,"tempting":[{"text":"Language filtering does distort corpora, so a filtering explanation sounds well grounded even where the sources are in English."}]}],"order":[1,2,0,3],"why":null},"prev":{"slug":"pretraining-finetuning-rlhf","title":"Pretraining, fine-tuning, and preference training","minutes":8},"next":{"slug":"the-loss-and-perplexity","title":"What the training objective actually measures","minutes":8}}}