Ninetieth percentile, against whom?
In its 2023 GPT-4 technical report, OpenAI described a simulated US bar exam result near the top 10% of test takers. A percentile is a comparison. It tells you where the score sits in a reference group; it is not the share of questions answered correctly.
Eric Martínez’s re-analysis found that approximate conversions from a February Illinois comparison, which included many repeat takers, could put the reported score near the 90th percentile. The July comparison gave an estimate around the 68th. Comparing with first-time July test takers gave around the 62nd. The paper also reports lower estimates when comparing only with passers, and identifies questions about how the essays were graded. The comparison and the scoring method belong beside the headline.
Quick check
Place a bet: against lawyers who passed the exam, where did the same score land?
What a headline may omit
Some items may already be familiar. In the GSM1k study, researchers wrote new grade-school maths problems to resemble an earlier benchmark. Some tested models scored lower on the new set. A gap is a clue to examine alongside difficulty, prompts and other evidence. It cannot establish how a particular example entered a model’s training.
The run may use help your product won’t have. Check which model version was tested, whether the prompt gave examples, which tools were allowed, whether multiple attempts were made and how the answer was scored.
The benchmark and your work may be different. A multiple-choice question can test something useful, while missing incomplete inputs, language differences or the consequences of an error in a classroom.
Run a fair pilot
This is a separate exercise from the chapter film’s room check. Here you compare two assistants on 20 fictional school notices with dates. Both assistants see the same source page and prompt for each notice. Decide what counts as a pass before looking at their answers: the date must match the source, and any missing date must be flagged instead of guessed.
| Result | Notices |
|---|---|
| Both pass | 12 |
| A alone passes | 2 |
| B alone passes | 5 |
| Neither passes | 1 |
A passes 14 of 20. B passes 17 of 20. That is a useful reason to inspect the seven cases where they differ. Check whether the misses cluster in a language, a type of notice or the most consequential situation. Twenty notices can reveal a problem; they cannot rule out rare failures. Use a fresh, larger set of examples before relying on the result for a more serious decision. NIST’s AI Risk Management Framework recommends test sets representative of expected conditions and documenting the method.
Ask the same questions of your own pilot and of a public benchmark: what was tested, under what conditions, against what comparison, and how much uncertainty remains?
Check your understanding
B passes 17 of 20 school notices and A passes 14. What can you conclude?
Pick the one you would defend. Your answer stays private.
Your place and first choices stay on this browser. Classroom credit is recorded separately.
Stuck on something?Ask Rukmini, your AI tutor