{"course":{"slug":"choosing-your-tools","title":"Choosing and Using the Tools","url":"https://addaly.com/learn/choosing-your-tools","level":"beginner","track":"work"},"lesson":{"slug":"reading-a-leaderboard","title":"Reading a leaderboard without being fooled","url":"https://addaly.com/learn/choosing-your-tools/reading-a-leaderboard","position":17,"of":76,"minutes":9,"answer":"Static benchmarks suffer contamination, saturation and best-case settings, arena votes reward length and confidence rather than correctness, and when a leaderboard disagrees with your own prompts, your prompts win.","keyIdea":"Static benchmarks suffer contamination, saturation and best-case settings, arena votes reward length and confidence rather than correctness, and when a leaderboard disagrees with your own prompts, your prompts win."},"body":{"format":"markdown","text":"## What the public scoreboards actually are\n\nThree kinds of number circulate, and they fail differently.\n\n**Static benchmarks.** A fixed set of questions with fixed answers: knowledge tests, maths sets, coding problems, graduate science questions. Reported as a percentage.\n\n**Arena rankings.** People are shown two anonymous answers to their own prompt and vote for one. Votes are converted into a chess-style rating, so a model's number reflects how often humans preferred it.\n\n**Vendor-reported evaluations.** The numbers in a launch post, produced by the company selling the model.\n\n## Four things that go wrong with static benchmarks\n\n**Contamination.** The questions are on the public web. Models are trained on the public web. When a benchmark's answers end up in training data, a high score can mean memorisation rather than capability. Labs try to filter this and cannot do it perfectly, which is why benchmarks that were meaningful three years ago are treated with suspicion now.\n\n**Saturation.** Once the top models all score above 90%, the remaining differences are mostly about the handful of questions that are ambiguous or wrongly labelled. Several widely quoted benchmarks are known to contain errors in a few per cent of their answers. A model can lose points by being right.\n\n**Favourable settings.** The reported number often comes from a best-case configuration: a particular prompt format, several attempts with the best kept, a high thinking budget. Two labs reporting on the same benchmark may not be running the same test at all. Read the footnotes; the good reports say exactly what they did.\n\n**It is not your work.** This is the big one. A benchmark of competition mathematics tells you about competition mathematics. Your job is drafting supplier emails in Gujarati, and the correlation is weaker than anyone would like.\n\n## What goes wrong with arena rankings\n\nThey measure *preference*, which is not the same as correctness. Human voters in a quick side-by-side reward answers that are longer, better formatted, more confident and more agreeable. The better arenas now publish a **style-controlled** ranking that statistically removes length and formatting effects, and the ordering changes noticeably when they do. If a leaderboard offers that view, it is the one to read.\n\nVoters also cannot check facts. In a thirty-second comparison, nobody verifies a citation. So a model that is confidently wrong in a well-organised way does well in an arena, which is exactly the failure mode you most need to avoid.\n\n## What leaderboards are good for\n\nThey are not useless, and dismissing them entirely is its own mistake.\n\n- **Tracking the field over years.** The trend line is real: capabilities have moved enormously and benchmarks record it.\n- **Spotting a newcomer.** If a lab you had not heard of appears near the top, that is worth ten minutes of your attention.\n- **Ruling things out.** A model far down the table on a task type close to yours is unlikely to surprise you.\n- **Cost-per-point comparisons.** Several sites plot price against score. That chart is more decision-relevant than the ranking itself, because it shows the cheap models sitting close to the expensive ones on ordinary tasks.\n\n## Independent evaluations, and why they are scarce\n\nA small number of organisations run evaluations they design themselves, hold back the questions, and publish the methodology. These are worth far more than a vendor's own numbers, and there are not many of them, because running one properly costs real money and annoys the companies being measured.\n\nWhen you find one, read what it says about its method before its results: whether the questions are held back, how many attempts each model got, whether the same prompt was used for every model, and who paid. A report that answers those four questions is worth more than a ranking with a bigger sample that answers none of them.\n\n## The rule of thumb\n\n**Gaps of a point or two: ignore.** Inside the noise, inside the contamination, inside the settings.\n\n**Gaps of ten points or more on a task type resembling yours: worth testing.** Not worth believing, worth testing.\n\n**Any gap at all, on a benchmark unlike your work: irrelevant to you.** Being second-best at graduate physics does not predict being second-best at your inbox.\n\nAnd when a leaderboard and your twelve prompts disagree, your twelve prompts win. They are a small, noisy sample of exactly the right thing, and the leaderboard is a large, careful sample of something else.\n\n> Figure: What a leaderboard gap is worth to you. A two by two grid, rows Gap of ten points or more and Gap of one or two points, columns Benchmark resembles your work and Benchmark unlike your work: Gap of ten points or more and Benchmark resembles your work: Test it; Gap of ten points or more and Benchmark unlike your work: Note it; Gap of one or two points and Benchmark resembles your work: Ignore; Gap of one or two points and Benchmark unlike your work: Ignore. Only the top-left cell changes anything, and it changes it into a test rather than a belief. Contamination, saturation and best-case settings all live comfortably inside a two-point gap, and when a leaderboard disagrees with your own twelve prompts, your prompts win."},"check":{"question":"An arena leaderboard offers a style-controlled view, and the ordering changes when you switch to it. What does that tell you about the default ranking?","options":[{"text":"The style-controlled view is an experimental adjustment and the default remains the more reliable measure of preference.","correct":false,"tempting":"Defends the default as more reliable when the adjustment exists precisely because the default conflates presentation with substance."},{"text":"Part of the default ordering came from answer length and formatting, not from content.","correct":true},{"text":"The default ranking is contaminated, because models that were trained on arena prompts score higher there.","correct":false,"tempting":"Borrows a static-benchmark failure; arena prompts are supplied live by voters, so the problem there is preference, not memorisation."},{"text":"The two views measure different model versions, and the ordering changed because style control uses a more recent snapshot.","correct":false,"tempting":"Invents a version difference; both views are computed from the same votes over the same models, differing only in the statistical adjustment applied."}]},"prev":{"slug":"scoring-without-fooling-yourself","title":"A rubric you can apply in ten seconds","url":"https://addaly.com/learn/choosing-your-tools/scoring-without-fooling-yourself"},"next":{"slug":"testing-refusals","title":"Testing the edges: refusals and house manners","url":"https://addaly.com/learn/choosing-your-tools/testing-refusals"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Reading a leaderboard without being fooled', in Choosing and Using the Tools. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/choosing-your-tools/reading-a-leaderboard (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Reading a leaderboard without being fooled","courseTitle":"Choosing and Using the Tools","module":{"index":2,"slug":"testing-it-yourself","title":"Testing a tool yourself"},"answer":null,"keyIdea":"Static benchmarks suffer contamination, saturation and best-case settings, arena votes reward length and confidence rather than correctness, and when a leaderboard disagrees with your own prompts, your prompts win.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["Benchmark resembles your work","Benchmark unlike your work"],"rows":["Gap of ten points or more","Gap of one or two points"],"type":"matrix","cells":[["Test it","Note it"],["Ignore","Ignore"]],"title":"What a leaderboard gap is worth to you","caption":"Only the top-left cell changes anything, and it changes it into a test rather than a belief. Contamination, saturation and best-case settings all live comfortably inside a two-point gap, and when a leaderboard disagrees with your own twelve prompts, your prompts win.","cellLabels":[["run your own twelve prompts","interesting, not about your work"],["inside the noise","irrelevant twice over"]]},"alt":"Figure: What a leaderboard gap is worth to you. A two by two grid, rows Gap of ten points or more and Gap of one or two points, columns Benchmark resembles your work and Benchmark unlike your work: Gap of ten points or more and Benchmark resembles your work: Test it; Gap of ten points or more and Benchmark unlike your work: Note it; Gap of one or two points and Benchmark resembles your work: Ignore; Gap of one or two points and Benchmark unlike your work: Ignore. Only the top-left cell changes anything, and it changes it into a test rather than a belief. Contamination, saturation and best-case settings all live comfortably inside a two-point gap, and when a leaderboard disagrees with your own twelve prompts, your prompts win."}},"blocks":[{"t":"h2","id":"what-the-public-scoreboards-actually-are","runs":[{"text":"What the public scoreboards actually are"}]},{"t":"p","runs":[{"text":"Three kinds of number circulate, and they fail differently."}]},{"t":"p","runs":[{"text":"Static benchmarks.","b":true},{"text":" A fixed set of questions with fixed answers: knowledge tests, maths sets, coding problems, graduate science questions. Reported as a percentage."}]},{"t":"p","runs":[{"text":"Arena rankings.","b":true},{"text":" People are shown two anonymous answers to their own prompt and vote for one. Votes are converted into a chess-style rating, so a model’s number reflects how often humans preferred it."}]},{"t":"p","runs":[{"text":"Vendor-reported evaluations.","b":true},{"text":" The numbers in a launch post, produced by the company selling the model."}]},{"t":"h2","id":"four-things-that-go-wrong-with-static-benchmarks","runs":[{"text":"Four things that go wrong with static benchmarks"}]},{"t":"p","runs":[{"text":"Contamination.","b":true},{"text":" The questions are on the public web. Models are trained on the public web. When a benchmark’s answers end up in training data, a high score can mean memorisation rather than capability. Labs try to filter this and cannot do it perfectly, which is why benchmarks that were meaningful three years ago are treated with suspicion now."}]},{"t":"p","runs":[{"text":"Saturation.","b":true},{"text":" Once the top models all score above 90%, the remaining differences are mostly about the handful of questions that are ambiguous or wrongly labelled. Several widely quoted benchmarks are known to contain errors in a few per cent of their answers. A model can lose points by being right."}]},{"t":"p","runs":[{"text":"Favourable settings.","b":true},{"text":" The reported number often comes from a best-case configuration: a particular prompt format, several attempts with the best kept, a high thinking budget. Two labs reporting on the same benchmark may not be running the same test at all. Read the footnotes; the good reports say exactly what they did."}]},{"t":"p","runs":[{"text":"It is not your work.","b":true},{"text":" This is the big one. A benchmark of competition mathematics tells you about competition mathematics. Your job is drafting supplier emails in Gujarati, and the correlation is weaker than anyone would like."}]},{"t":"h2","id":"what-goes-wrong-with-arena-rankings","runs":[{"text":"What goes wrong with arena rankings"}]},{"t":"p","runs":[{"text":"They measure "},{"text":"preference","i":true},{"text":", which is not the same as correctness. Human voters in a quick side-by-side reward answers that are longer, better formatted, more confident and more agreeable. The better arenas now publish a "},{"text":"style-controlled","b":true},{"text":" ranking that statistically removes length and formatting effects, and the ordering changes noticeably when they do. If a leaderboard offers that view, it is the one to read."}]},{"t":"p","runs":[{"text":"Voters also cannot check facts. In a thirty-second comparison, nobody verifies a citation. So a model that is confidently wrong in a well-organised way does well in an arena, which is exactly the failure mode you most need to avoid."}]},{"t":"h2","id":"what-leaderboards-are-good-for","runs":[{"text":"What leaderboards are good for"}]},{"t":"p","runs":[{"text":"They are not useless, and dismissing them entirely is its own mistake."}]},{"t":"ul","items":[[{"text":"Tracking the field over years.","b":true},{"text":" The trend line is real: capabilities have moved enormously and benchmarks record it."}],[{"text":"Spotting a newcomer.","b":true},{"text":" If a lab you had not heard of appears near the top, that is worth ten minutes of your attention."}],[{"text":"Ruling things out.","b":true},{"text":" A model far down the table on a task type close to yours is unlikely to surprise you."}],[{"text":"Cost-per-point comparisons.","b":true},{"text":" Several sites plot price against score. That chart is more decision-relevant than the ranking itself, because it shows the cheap models sitting close to the expensive ones on ordinary tasks."}]]},{"t":"h2","id":"independent-evaluations-and-why-they-are-scarce","runs":[{"text":"Independent evaluations, and why they are scarce"}]},{"t":"p","runs":[{"text":"A small number of organisations run evaluations they design themselves, hold back the questions, and publish the methodology. These are worth far more than a vendor’s own numbers, and there are not many of them, because running one properly costs real money and annoys the companies being measured."}]},{"t":"p","runs":[{"text":"When you find one, read what it says about its method before its results: whether the questions are held back, how many attempts each model got, whether the same prompt was used for every model, and who paid. A report that answers those four questions is worth more than a ranking with a bigger sample that answers none of them."}]},{"t":"h2","id":"the-rule-of-thumb","runs":[{"text":"The rule of thumb"}]},{"t":"p","runs":[{"text":"Gaps of a point or two: ignore.","b":true},{"text":" Inside the noise, inside the contamination, inside the settings."}]},{"t":"p","runs":[{"text":"Gaps of ten points or more on a task type resembling yours: worth testing.","b":true},{"text":" Not worth believing, worth testing."}]},{"t":"p","runs":[{"text":"Any gap at all, on a benchmark unlike your work: irrelevant to you.","b":true},{"text":" Being second-best at graduate physics does not predict being second-best at your inbox."}]},{"t":"p","runs":[{"text":"And when a leaderboard and your twelve prompts disagree, your twelve prompts win. They are a small, noisy sample of exactly the right thing, and the leaderboard is a large, careful sample of something else."}]}],"check":{"question":[{"text":"An arena leaderboard offers a style-controlled view, and the ordering changes when you switch to it. What does that tell you about the default ranking?"}],"options":[{"text":[{"text":"The style-controlled view is an experimental adjustment and the default remains the more reliable measure of preference."}],"correct":false,"tempting":[{"text":"Defends the default as more reliable when the adjustment exists precisely because the default conflates presentation with substance."}]},{"text":[{"text":"Part of the default ordering came from answer length and formatting, not from content."}],"correct":true,"tempting":null},{"text":[{"text":"The default ranking is contaminated, because models that were trained on arena prompts score higher there."}],"correct":false,"tempting":[{"text":"Borrows a static-benchmark failure; arena prompts are supplied live by voters, so the problem there is preference, not memorisation."}]},{"text":[{"text":"The two views measure different model versions, and the ordering changed because style control uses a more recent snapshot."}],"correct":false,"tempting":[{"text":"Invents a version difference; both views are computed from the same votes over the same models, differing only in the statistical adjustment applied."}]}],"order":[0,2,3,1],"why":null},"prev":{"slug":"scoring-without-fooling-yourself","title":"A rubric you can apply in ten seconds","minutes":7},"next":{"slug":"testing-refusals","title":"Testing the edges: refusals and house manners","minutes":8}}}