AI, Safety and What Goes WrongLesson 3 of 73

Module 1 · How these systems fail

Where the data came from

In this lesson

  • 3 min read
  • 4 reading sections
  • Interactive activity
  • Knowledge check
  • Lesson 3 of 9 in this chapter

This chapter’s goalSort a failure into design, capability or misuse, read a benchmark or a demo for what it leaves out, and set how much checking a use needs by what it costs if wrong.

This chapter’s media

The film explains chapter ideas. The audiobook narrates the complete teaching text.

FormatEnglishहिंदी
Chapter filmWatch2:47Not available
Full audiobookNot availableNot available
GPT-3, 2020: how many times each source was seenWikipedia, 3 billiontokens3.4WebText2, 19 billiontokens2.9Books1, 12 billion tokens1.9Common Crawl, filtered,410 billion tokens0.44Books2, 55 billion tokens0.43times seen in trainingFrom Table 2.2 of the GPT-3 paper. The team sampled by judged quality, not by size, so an encyclopediashaped the model far more per word than the filtered web did.
From Table 2.2 of the GPT-3 paper. The team sampled by judged quality, not by size, so an encyclopedia shaped the model far more per word than the filtered web did.

Figure

A little at a time

Explore an idea, try it, then check what stays.

Step 1 of 7Free · no account needed
Lesson steps

Forty-five terabytes in, 570 gigabytes kept

In 2020 OpenAI published the recipe for GPT-3, the model family ChatGPT grew from. Its largest ingredient was Common Crawl, a free archive of the web. The team took 41 monthly snapshots from 2016 to 2019: 45 terabytes of compressed text. After filtering, they kept 570 gigabytes. Roughly one part in 80 survived.

Your place and first choices stay on this browser. Classroom credit is recorded separately.

Stuck on something?Ask Rukmini, your AI tutor

How to cite: Addaly (2026) ‘Where the data came from’, in AI, Safety and What Goes Wrong. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-and-you/training-data-is-a-choice (Accessed: 21 November 2026).