Forty-five terabytes in, 570 gigabytes kept
In 2020 OpenAI published the recipe for GPT-3, the model family ChatGPT grew from. Its largest ingredient was Common Crawl, a free archive of the web. The team took 41 monthly snapshots from 2016 to 2019: 45 terabytes of compressed text. After filtering, they kept 570 gigabytes. Roughly one part in 80 survived.
Quick check
Your call: who decided which part of the web to keep?
“Trained on the internet” sounds like a fact of nature, like rain. It describes a chain of decisions, and the first one was which part in 80 to keep.
How “good” was defined
The team trained a classifier to tell high-quality pages from raw crawl. For its examples of quality they used WebText, a collection of pages linked from Reddit posts, plus Wikipedia and a books collection. Pages that resembled those were kept.
So “good writing” meant, in practice, “looks like what one English-speaking online community linked to, plus an encyclopedia”. Every answer the model gives inherits that definition.
How often each source was seen
Next the team chose how often to show each source. Their paper says the mix was deliberately not in proportion to size.
Wikipedia, 3 billion tokens, was seen 3.4 times. The filtered web, 410 billion tokens, was seen 0.44 times: more than half of it never came up at all.
What the choices do to you
Language. By word count, 93% of GPT-3’s data was English. A model that answers well in English and worse in Marathi or Bengali saw far less of those languages.
Time. Every model has a cutoff. Ask about something after it and you get a refusal, a search result, or a confident answer built from the shape of older events. The third is the dangerous one.
Who writes online. Web text over-represents people with the time, connection and confidence to publish. The model’s sense of what is normal comes from them.
Where you can look
Some training sets are public. C4, The Pile, Dolma and FineWeb publish what they contain, and some let you search for a document. For most commercial models you cannot see the mix. The EU AI Act now requires providers of general-purpose models to publish a sufficiently detailed summary of the content used for training.
When someone says a model was “trained on all of human knowledge”, the useful reply is to ask which crawl it came from, how it was filtered, in what proportions it was mixed, and up to what date.
Check your understanding
An assistant gives noticeably weaker answers in Bengali than in English, on a subject where good Bengali material exists. What is the likeliest explanation?
Pick the one you would defend. Your answer stays private.
Your place and first choices stay on this browser. Classroom credit is recorded separately.
Stuck on something?Ask Rukmini, your AI tutor