AI, Safety and What Goes WrongLesson 4 of 73

Module 1 · How these systems fail

What a benchmark score leaves out

In this lesson

  • 3 min read
  • 4 reading sections
  • Interactive activity
  • Knowledge check
  • Lesson 4 of 9 in this chapter

This chapter’s goalSort a failure into design, capability or misuse, read a benchmark or a demo for what it leaves out, and set how much checking a use needs by what it costs if wrong.

This chapter’s media

The film explains chapter ideas. The audiobook narrates the complete teaching text.

FormatEnglishहिंदी
Chapter filmWatch2:47Not available
Full audiobookNot availableNot available
The same reported score against three groupsFebruary Illinoiscomparison90July comparison, overall68July comparison,first-time takers62approx. percentileApproximate percentile estimates from Martínez (2024), section 3.2.1. These show a relative positionwithin each comparison group, not the percentage of questions answered correctly.
Approximate percentile estimates from Martínez (2024), section 3.2.1. These show a relative position within each comparison group, not the percentage of questions answered correctly.

Figure

A little at a time

Explore an idea, try it, then check what stays.

Step 1 of 7Free · no account needed
Lesson steps

Ninetieth percentile, against whom?

In its 2023 GPT-4 technical report, OpenAI described a simulated US bar exam result near the top 10% of test takers. A percentile is a comparison. It tells you where the score sits in a reference group; it is not the share of questions answered correctly.

Eric Martínez’s re-analysis found that approximate conversions from a February Illinois comparison, which included many repeat takers, could put the reported score near the 90th percentile. The July comparison gave an estimate around the 68th. Comparing with first-time July test takers gave around the 62nd. The paper also reports lower estimates when comparing only with passers, and identifies questions about how the essays were graded. The comparison and the scoring method belong beside the headline.

Your place and first choices stay on this browser. Classroom credit is recorded separately.

Stuck on something?Ask Rukmini, your AI tutor

How to cite: Addaly (2026) ‘What a benchmark score leaves out’, in AI, Safety and What Goes Wrong. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-and-you/the-benchmark-and-the-world (Accessed: 21 November 2026).