Public benchmarks, and what they are for

The numbers on the model card

Every model launch comes with a table: MMLU, GSM8K, HumanEval, MATH, SWE-bench, GPQA, and a position on a leaderboard. It is worth knowing what these actually measure, because they get quoted in decisions they cannot support.

• MMLU: about 16,000 multiple-choice questions across 57 school and university subjects. Four options, one correct. It measures recall of academic knowledge in a format nothing in your product resembles.
• GSM8K: 8,500 grade-school arithmetic word problems. Largely saturated: frontier models score above 95%, so differences are noise plus contamination.
• HumanEval: 164 small Python functions, each with unit tests. A model writes the body, the tests run. Real and verifiable, and 164 items is a very small set with a wide interval.
• SWE-bench: real GitHub issues from real repositories, scored by whether the patch makes the project’s tests pass. Much closer to actual work, and much harder to game.
• LMArena: humans compare two anonymous responses and pick one, aggregated into an Elo rating.

The variation in construction matters. A benchmark scored by running tests is a different kind of evidence from one scored by matching a letter.

Contamination

These sets are on the public web. Models are trained on the public web. A model may have seen the questions and the answers during training, and no one outside the lab can verify that it did not.

The signs are recognisable once you know them: a suspiciously high score on an old benchmark alongside ordinary performance on a freshly written version of the same task; large sensitivity to trivial rewording that should not change difficulty; scores that fall sharply on a benchmark released after the model’s training cutoff. Some benchmark authors embed canary strings: unique identifiers that should never appear in training data, so contamination can be detected afterwards. Almost nobody checks.

The practical consequence is not “benchmarks are fake”. It is that a benchmark’s absolute number means less than the difference between two models on it, and a two-point gap on MMLU means nothing at all.

The leaderboard has its own biases

Arena-style rankings are more robust than static benchmarks because the questions keep changing, but they measure a specific thing: which answer a volunteer prefers, after reading both, in a chat window, usually within thirty seconds.

That rewards formatting. Answers with headings, bullets and a summary line win against denser correct prose. It rewards length, up to a point. It rewards a confident tone, which is exactly the property you do not want in a system that must sometimes say it does not know. And the prompts come from people who volunteer to test chatbots, which is not your user base.

Style-controlled variants of these rankings exist and shuffle the order noticeably, which is itself the evidence that formatting was doing real work in the original.

What benchmarks are legitimately for

They are a shortlist, and that is a genuine use.

You cannot try forty models. Public scores, price and context window narrow it to three or four worth an afternoon each. A model that is far down on every coding benchmark is unlikely to surprise you on your coding task. A model with strong multilingual scores is a better first guess for Hindi support than one without.

What they cannot do is decide. The gap between benchmark rank and your task is enormous: your inputs are messier, your criteria are yours, your prompt does most of the work, and your latency and cost limits may exclude the top model anyway. Cases where the second- or third-ranked model wins on the actual task are common, usually for a boring reason: it follows format instructions more reliably, or it refuses less often on legitimate content in your domain.

The honest procedure

1. Shortlist three models from public scores, price and context length. Fifteen minutes.
2. Run your own hundred items against all three, same prompt, same day. An afternoon.
3. Compare on your metric, with intervals, plus cost and p95 latency.
4. Pin the winner by exact version string and record why. That record is what saves you when the model is deprecated in eight months.

Step two is the whole course. Steps one and four are why this lesson exists.

One last thing worth internalising: when a vendor claims a benchmark lead, the interesting question is not whether the number is true. It usually is. The question is whether that benchmark’s task resembles yours, and for most products the honest answer is that it does not resemble it much at all.