Reading a leaderboard without being fooled

What the public scoreboards actually are

Three kinds of number circulate, and they fail differently.

Static benchmarks. A fixed set of questions with fixed answers: knowledge tests, maths sets, coding problems, graduate science questions. Reported as a percentage.

Arena rankings. People are shown two anonymous answers to their own prompt and vote for one. Votes are converted into a chess-style rating, so a model’s number reflects how often humans preferred it.

Vendor-reported evaluations. The numbers in a launch post, produced by the company selling the model.

Four things that go wrong with static benchmarks

Contamination. The questions are on the public web. Models are trained on the public web. When a benchmark’s answers end up in training data, a high score can mean memorisation rather than capability. Labs try to filter this and cannot do it perfectly, which is why benchmarks that were meaningful three years ago are treated with suspicion now.

Saturation. Once the top models all score above 90%, the remaining differences are mostly about the handful of questions that are ambiguous or wrongly labelled. Several widely quoted benchmarks are known to contain errors in a few per cent of their answers. A model can lose points by being right.

Favourable settings. The reported number often comes from a best-case configuration: a particular prompt format, several attempts with the best kept, a high thinking budget. Two labs reporting on the same benchmark may not be running the same test at all. Read the footnotes; the good reports say exactly what they did.

It is not your work. This is the big one. A benchmark of competition mathematics tells you about competition mathematics. Your job is drafting supplier emails in Gujarati, and the correlation is weaker than anyone would like.

What goes wrong with arena rankings

They measure preference, which is not the same as correctness. Human voters in a quick side-by-side reward answers that are longer, better formatted, more confident and more agreeable. The better arenas now publish a style-controlled ranking that statistically removes length and formatting effects, and the ordering changes noticeably when they do. If a leaderboard offers that view, it is the one to read.

Voters also cannot check facts. In a thirty-second comparison, nobody verifies a citation. So a model that is confidently wrong in a well-organised way does well in an arena, which is exactly the failure mode you most need to avoid.

What leaderboards are good for

They are not useless, and dismissing them entirely is its own mistake.

• Tracking the field over years. The trend line is real: capabilities have moved enormously and benchmarks record it.
• Spotting a newcomer. If a lab you had not heard of appears near the top, that is worth ten minutes of your attention.
• Ruling things out. A model far down the table on a task type close to yours is unlikely to surprise you.
• Cost-per-point comparisons. Several sites plot price against score. That chart is more decision-relevant than the ranking itself, because it shows the cheap models sitting close to the expensive ones on ordinary tasks.

Independent evaluations, and why they are scarce

A small number of organisations run evaluations they design themselves, hold back the questions, and publish the methodology. These are worth far more than a vendor’s own numbers, and there are not many of them, because running one properly costs real money and annoys the companies being measured.

When you find one, read what it says about its method before its results: whether the questions are held back, how many attempts each model got, whether the same prompt was used for every model, and who paid. A report that answers those four questions is worth more than a ranking with a bigger sample that answers none of them.

The rule of thumb

Gaps of a point or two: ignore. Inside the noise, inside the contamination, inside the settings.

Gaps of ten points or more on a task type resembling yours: worth testing. Not worth believing, worth testing.

Any gap at all, on a benchmark unlike your work: irrelevant to you. Being second-best at graduate physics does not predict being second-best at your inbox.

And when a leaderboard and your twelve prompts disagree, your twelve prompts win. They are a small, noisy sample of exactly the right thing, and the leaderboard is a large, careful sample of something else.