The leaderboards that still carry information

What kills a leaderboard

Two things, and both are predictable. Saturation: everybody scores above 85 and the remaining spread is the benchmark’s own error rate. Contamination: the test set is public, so it is in the training data, and the ranking measures exposure rather than capability. Hugging Face’s Open LLM Leaderboard was eventually archived rather than patched, which is the clearest statement anyone has made about how far both had gone.

So the question is not which leaderboard is best, but which design properties resist those two failures. There are four.

A private or rotating test set. Items that cannot be trained on because they were not published, or were published after the model was trained.

Execution-based verification. Code that must compile and pass tests, or a task that must actually complete. Something has to work, which is far harder to fake than picking a letter.

Fresh human prompts. People type new questions, so there is nothing to memorise, at the cost of measuring preference rather than correctness.

A published harness. If you can rerun it yourself on a model you understand, you can calibrate the numbers.

The ones worth reading, and what each measures

Pairwise human preference arenas. Two anonymous answers, a person picks one, ratings computed with a Bradley-Terry or Elo-style model. Resistant to contamination because the prompts are new. Measures preference, which rewards length, headers, confident tone and agreement with the asker, which is why the style-controlled variant of the ranking is the one to read. Note also that models are often tested under codenames, and only results a lab likes get claimed publicly.

Rotating-question benchmarks. Evaluations that add new items monthly and score models only on items published after their training cut-off. This directly targets contamination, and it is the most useful recent development in public evaluation.

Execution-based code and agent benchmarks. Tasks against real repositories where a patch must make failing tests pass. Expensive to run, hard to fake, and the closest public proxy for whether a model can do work.

Domain leaderboards: embeddings, retrieval, speech recognition, translation, long context. Usually narrower and often more honest, because the communities are smaller and the metrics are older and better understood. Each has its own contamination story worth reading before quoting.

Price and speed trackers. Independent measurement of tokens per second, time to first token and price across providers. These measure a real and stable thing, and they answer a different question from quality, which, given the earlier lesson on providers serving the same model differently, is a question you actually need answered.

Reading a ranking properly

Look for the interval. Preference ratings come with confidence intervals, and adjacent models routinely overlap. A 15-point gap with overlapping intervals is not a ranking, it is a tie displayed in an order.

Check the sample size behind each entry. A model with 800 votes and one with 40,000 are not measured to the same precision.

Check the date. A three-month-old leaderboard in this field describes a different landscape.

Check what was actually run. Which quantisation, which context length, which provider: the same warnings from the providers lesson apply to leaderboard entries served through an API.

What no leaderboard can do

None of them evaluates your task, your documents, your languages, your latency budget or your licence constraint. The correct use of a leaderboard is shortlisting: it narrows thirty candidates to three that are plausibly in the right class. The choice between those three is made on forty of your own examples, and that measurement outranks every public number.

The one-sentence test for whether a leaderboard deserves your attention: can you tell, from its documentation, exactly what was run against what, and could you reproduce one row of it? If not, it is entertainment.