The leaderboard is not measuring your task

The number that moved you

A model appears at the top of a table. You pick it. Two weeks later it is worse at your actual work than the boring model you were already using, and you cannot say why.

Leaderboards on the Hub come and go — the well-known Open LLM Leaderboard ran for years and was archived in 2025 — and there are focused ones for embeddings, speech, code, vision and individual languages. Their failure modes outlive any particular table, so learn the failure modes.

Contamination

Benchmarks are published so that people can use them. That means the test questions and their answers are on the public web. Models are pretrained on scrapes of the public web. So a benchmark's answers can end up in a model's training data, and a model that has memorised the answers scores like a model that can reason.

This is usually not cheating. It is a structural consequence of training on the internet the benchmark lives on, and it gets worse the longer a benchmark has existed. Two signs to look for:

• A large gap between a famous public benchmark and a private or freshly written one on the same skill.
• A high score on an old benchmark next to an ordinary score on a benchmark released after that model's training data cutoff.

This is why benchmarks with hidden test sets and rolling questions exist, and why a score on a benchmark from three years ago tells you less every year.

Selective reporting

A card shows the benchmarks the model won. Nobody publishes the six they lost. When you see five results, ask which of the standard set for that task is absent. The gaps are information — often better information than the numbers, because the numbers are chosen and the gaps are not.

Preference arenas and the style problem

Human preference arenas show two anonymous answers and count votes. That measures something real and something no automatic benchmark captures. It also measures formatting, length, confidence and agreeableness.

A model that answers in tidy headed bullet points, hedges nothing and writes four hundred words tends to beat a model that is correct in two sentences. A model tuned to agree with the user beats one that says the premise is wrong. So an arena rank is a measurement of likeability under one style of interaction, which correlates with usefulness and is not the same thing. Read it as such.

The gap that actually matters

The famous general benchmarks are, overwhelmingly, English multiple-choice questions about general knowledge. Consider what your work is:

• Triaging complaints written in Hindi with English words mixed in.
• Transcribing Nigerian-accented English over a noisy phone line.
• Pulling five fields out of a scanned Bengali invoice.
• Rewriting product descriptions in a house style, without inventing features.

For none of these does the leaderboard rank predict much. Multilingual scores are usually an average across many languages, and an average is exactly the statistic that hides the fact that your language is one of the ones that fails. Worse, a model heavily tuned for benchmark-style answers can be measurably *worse* at following a plain instruction in your format, because that tuning taught it to produce essays and lettered options.

The evaluation you can build this afternoon

The fix is small and unglamorous, and it beats every leaderboard for your decision because it measures the thing you are actually buying.

1. 1Collect thirty real examples from your own work. Not invented ones — real ones, including the awkward ones.
2. 2Write down, for each, the output you would accept. Where several outputs are acceptable, write the rule.
3. 3Run each candidate model over all thirty. Free Spaces and provider free credits are enough for this.
4. 4Score by hand, in a spreadsheet, one row per example and one column per model.
5. 5Read the failures rather than the total. The pattern in what breaks is the finding.

That is half a day, and you can rerun it every time you consider switching. /learn/evaluating-ai sets out the method in full, including how to keep a set like this honest over time.

The rule that follows

Use the leaderboard to get from four hundred candidates to five. Use your thirty examples to get from five to one. Filtering the Hub by task and sorting by downloads is a reasonable third filter — popularity is not quality, but a heavily used model is one whose bugs somebody else has already hit and written up in the Community tab.

Do this now

Open the eval table of a model you are considering and name one standard benchmark for its task that is missing from the table. Then write down five real examples from your own work. Twenty-five to go.