{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"the-benchmark-you-read-about","title":"Public benchmarks, and what they are for","url":"https://addaly.com/learn/evaluating-ai/the-benchmark-you-read-about","position":20,"of":76,"minutes":8,"answer":"Public benchmarks are a shortlist of models worth testing, never a decision, because contamination, format and their distance from your task all sit between the score and your product.","keyIdea":"Public benchmarks are a shortlist of models worth testing, never a decision, because contamination, format and their distance from your task all sit between the score and your product."},"body":{"format":"markdown","text":"## The numbers on the model card\n\nEvery model launch comes with a table: MMLU, GSM8K, HumanEval, MATH, SWE-bench, GPQA, and a position on a leaderboard. It is worth knowing what these actually measure, because they get quoted in decisions they cannot support.\n\n- **MMLU**: about 16,000 multiple-choice questions across 57 school and university subjects. Four options, one correct. It measures recall of academic knowledge in a format nothing in your product resembles.\n- **GSM8K**: 8,500 grade-school arithmetic word problems. Largely saturated: frontier models score above 95%, so differences are noise plus contamination.\n- **HumanEval**: 164 small Python functions, each with unit tests. A model writes the body, the tests run. Real and verifiable, and 164 items is a very small set with a wide interval.\n- **SWE-bench**: real GitHub issues from real repositories, scored by whether the patch makes the project's tests pass. Much closer to actual work, and much harder to game.\n- **LMArena**: humans compare two anonymous responses and pick one, aggregated into an Elo rating.\n\nThe variation in construction matters. A benchmark scored by running tests is a different kind of evidence from one scored by matching a letter.\n\n## Contamination\n\nThese sets are on the public web. Models are trained on the public web. A model may have seen the questions and the answers during training, and no one outside the lab can verify that it did not.\n\nThe signs are recognisable once you know them: a suspiciously high score on an old benchmark alongside ordinary performance on a freshly written version of the same task; large sensitivity to trivial rewording that should not change difficulty; scores that fall sharply on a benchmark released after the model's training cutoff. Some benchmark authors embed *canary strings*: unique identifiers that should never appear in training data, so contamination can be detected afterwards. Almost nobody checks.\n\nThe practical consequence is not \"benchmarks are fake\". It is that a benchmark's absolute number means less than the difference between two models on it, and a two-point gap on MMLU means nothing at all.\n\n## The leaderboard has its own biases\n\nArena-style rankings are more robust than static benchmarks because the questions keep changing, but they measure a specific thing: which answer a volunteer prefers, after reading both, in a chat window, usually within thirty seconds.\n\nThat rewards formatting. Answers with headings, bullets and a summary line win against denser correct prose. It rewards length, up to a point. It rewards a confident tone, which is exactly the property you do not want in a system that must sometimes say it does not know. And the prompts come from people who volunteer to test chatbots, which is not your user base.\n\nStyle-controlled variants of these rankings exist and shuffle the order noticeably, which is itself the evidence that formatting was doing real work in the original.\n\n## What benchmarks are legitimately for\n\nThey are a **shortlist**, and that is a genuine use.\n\nYou cannot try forty models. Public scores, price and context window narrow it to three or four worth an afternoon each. A model that is far down on every coding benchmark is unlikely to surprise you on your coding task. A model with strong multilingual scores is a better first guess for Hindi support than one without.\n\nWhat they cannot do is decide. The gap between benchmark rank and your task is enormous: your inputs are messier, your criteria are yours, your prompt does most of the work, and your latency and cost limits may exclude the top model anyway. Cases where the second- or third-ranked model wins on the actual task are common, usually for a boring reason: it follows format instructions more reliably, or it refuses less often on legitimate content in your domain.\n\n## The honest procedure\n\n1. Shortlist three models from public scores, price and context length. Fifteen minutes.\n2. Run your own hundred items against all three, same prompt, same day. An afternoon.\n3. Compare on your metric, with intervals, plus cost and p95 latency.\n4. Pin the winner by exact version string and record why. That record is what saves you when the model is deprecated in eight months.\n\nStep two is the whole course. Steps one and four are why this lesson exists.\n\nOne last thing worth internalising: when a vendor claims a benchmark lead, the interesting question is not whether the number is true. It usually is. The question is whether that benchmark's task resembles yours, and for most products the honest answer is that it does not resemble it much at all."},"check":{"question":"Model B beats Model A by 2.1 points on MMLU and sits three places higher on a public leaderboard. Your team builds an invoice-extraction feature. What follows?","options":[{"text":"Model B is the better general model, so it should be the default and A kept only as a fallback.","correct":false,"tempting":"Treats a small aggregate lead on academic multiple-choice as a transferable ranking for a structured-extraction task."},{"text":"Both are worth an afternoon on your own hundred invoices; the benchmark gap is too small and too distant from extraction to decide anything.","correct":true},{"text":"Leaderboard position is the stronger signal because human preference is closer to real use than a multiple-choice test.","correct":false,"tempting":"Prefers the ranking that rewards chat formatting and confident tone, neither of which is relevant to extracting fields from a document."},{"text":"Neither number is usable because both benchmarks are contaminated, so public evaluation should be ignored entirely.","correct":false,"tempting":"Overcorrects into dismissing benchmarks altogether, discarding their real use as a way to narrow forty candidates to three."}]},"prev":{"slug":"evaluating-rag","title":"Three questions, not one, for a retrieval system","url":"https://addaly.com/learn/evaluating-ai/evaluating-rag"},"next":{"slug":"simulating-the-null","title":"Asking what pure chance would have produced","url":"https://addaly.com/learn/evaluating-ai/simulating-the-null"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Public benchmarks, and what they are for', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/the-benchmark-you-read-about (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Public benchmarks, and what they are for","courseTitle":"Knowing If It Works","module":{"index":2,"slug":"metrics-that-mean-something","title":"Metrics that mean something"},"answer":null,"keyIdea":"Public benchmarks are a shortlist of models worth testing, never a decision, because contamination, format and their distance from your task all sit between the score and your product.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-numbers-on-the-model-card","runs":[{"text":"The numbers on the model card"}]},{"t":"p","runs":[{"text":"Every model launch comes with a table: MMLU, GSM8K, HumanEval, MATH, SWE-bench, GPQA, and a position on a leaderboard. It is worth knowing what these actually measure, because they get quoted in decisions they cannot support."}]},{"t":"ul","items":[[{"text":"MMLU","b":true},{"text":": about 16,000 multiple-choice questions across 57 school and university subjects. Four options, one correct. It measures recall of academic knowledge in a format nothing in your product resembles."}],[{"text":"GSM8K","b":true},{"text":": 8,500 grade-school arithmetic word problems. Largely saturated: frontier models score above 95%, so differences are noise plus contamination."}],[{"text":"HumanEval","b":true},{"text":": 164 small Python functions, each with unit tests. A model writes the body, the tests run. Real and verifiable, and 164 items is a very small set with a wide interval."}],[{"text":"SWE-bench","b":true},{"text":": real GitHub issues from real repositories, scored by whether the patch makes the project’s tests pass. Much closer to actual work, and much harder to game."}],[{"text":"LMArena","b":true},{"text":": humans compare two anonymous responses and pick one, aggregated into an Elo rating."}]]},{"t":"p","runs":[{"text":"The variation in construction matters. A benchmark scored by running tests is a different kind of evidence from one scored by matching a letter."}]},{"t":"h2","id":"contamination","runs":[{"text":"Contamination"}]},{"t":"p","runs":[{"text":"These sets are on the public web. Models are trained on the public web. A model may have seen the questions and the answers during training, and no one outside the lab can verify that it did not."}]},{"t":"p","runs":[{"text":"The signs are recognisable once you know them: a suspiciously high score on an old benchmark alongside ordinary performance on a freshly written version of the same task; large sensitivity to trivial rewording that should not change difficulty; scores that fall sharply on a benchmark released after the model’s training cutoff. Some benchmark authors embed "},{"text":"canary strings","i":true},{"text":": unique identifiers that should never appear in training data, so contamination can be detected afterwards. Almost nobody checks."}]},{"t":"p","runs":[{"text":"The practical consequence is not “benchmarks are fake”. It is that a benchmark’s absolute number means less than the difference between two models on it, and a two-point gap on MMLU means nothing at all."}]},{"t":"h2","id":"the-leaderboard-has-its-own-biases","runs":[{"text":"The leaderboard has its own biases"}]},{"t":"p","runs":[{"text":"Arena-style rankings are more robust than static benchmarks because the questions keep changing, but they measure a specific thing: which answer a volunteer prefers, after reading both, in a chat window, usually within thirty seconds."}]},{"t":"p","runs":[{"text":"That rewards formatting. Answers with headings, bullets and a summary line win against denser correct prose. It rewards length, up to a point. It rewards a confident tone, which is exactly the property you do not want in a system that must sometimes say it does not know. And the prompts come from people who volunteer to test chatbots, which is not your user base."}]},{"t":"p","runs":[{"text":"Style-controlled variants of these rankings exist and shuffle the order noticeably, which is itself the evidence that formatting was doing real work in the original."}]},{"t":"h2","id":"what-benchmarks-are-legitimately-for","runs":[{"text":"What benchmarks are legitimately for"}]},{"t":"p","runs":[{"text":"They are a "},{"text":"shortlist","b":true},{"text":", and that is a genuine use."}]},{"t":"p","runs":[{"text":"You cannot try forty models. Public scores, price and context window narrow it to three or four worth an afternoon each. A model that is far down on every coding benchmark is unlikely to surprise you on your coding task. A model with strong multilingual scores is a better first guess for Hindi support than one without."}]},{"t":"p","runs":[{"text":"What they cannot do is decide. The gap between benchmark rank and your task is enormous: your inputs are messier, your criteria are yours, your prompt does most of the work, and your latency and cost limits may exclude the top model anyway. Cases where the second- or third-ranked model wins on the actual task are common, usually for a boring reason: it follows format instructions more reliably, or it refuses less often on legitimate content in your domain."}]},{"t":"h2","id":"the-honest-procedure","runs":[{"text":"The honest procedure"}]},{"t":"ol","start":1,"items":[[{"text":"Shortlist three models from public scores, price and context length. Fifteen minutes."}],[{"text":"Run your own hundred items against all three, same prompt, same day. An afternoon."}],[{"text":"Compare on your metric, with intervals, plus cost and p95 latency."}],[{"text":"Pin the winner by exact version string and record why. That record is what saves you when the model is deprecated in eight months."}]]},{"t":"p","runs":[{"text":"Step two is the whole course. Steps one and four are why this lesson exists."}]},{"t":"p","runs":[{"text":"One last thing worth internalising: when a vendor claims a benchmark lead, the interesting question is not whether the number is true. It usually is. The question is whether that benchmark’s task resembles yours, and for most products the honest answer is that it does not resemble it much at all."}]}],"check":{"question":[{"text":"Model B beats Model A by 2.1 points on MMLU and sits three places higher on a public leaderboard. Your team builds an invoice-extraction feature. What follows?"}],"options":[{"text":[{"text":"Model B is the better general model, so it should be the default and A kept only as a fallback."}],"correct":false,"tempting":[{"text":"Treats a small aggregate lead on academic multiple-choice as a transferable ranking for a structured-extraction task."}]},{"text":[{"text":"Both are worth an afternoon on your own hundred invoices; the benchmark gap is too small and too distant from extraction to decide anything."}],"correct":true,"tempting":null},{"text":[{"text":"Leaderboard position is the stronger signal because human preference is closer to real use than a multiple-choice test."}],"correct":false,"tempting":[{"text":"Prefers the ranking that rewards chat formatting and confident tone, neither of which is relevant to extracting fields from a document."}]},{"text":[{"text":"Neither number is usable because both benchmarks are contaminated, so public evaluation should be ignored entirely."}],"correct":false,"tempting":[{"text":"Overcorrects into dismissing benchmarks altogether, discarding their real use as a way to narrow forty candidates to three."}]}],"order":[2,3,1,0],"why":null},"prev":{"slug":"evaluating-rag","title":"Three questions, not one, for a retrieval system","minutes":10},"next":{"slug":"simulating-the-null","title":"Asking what pure chance would have produced","minutes":9}}}