{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"leaderboards-that-still-work","title":"The leaderboards that still carry information","url":"https://addaly.com/learn/open-models/leaderboards-that-still-work","position":64,"of":71,"minutes":9,"answer":"Leaderboards die from saturation and contamination, so the ones that still carry information use private or rotating items, execution-based verification or fresh human prompts, and even then their proper use is shortlisting, not deciding.","keyIdea":"Leaderboards die from saturation and contamination, so the ones that still carry information use private or rotating items, execution-based verification or fresh human prompts, and even then their proper use is shortlisting, not deciding."},"body":{"format":"markdown","text":"## What kills a leaderboard\n\nTwo things, and both are predictable. **Saturation**: everybody scores above 85 and the remaining spread is the benchmark's own error rate. **Contamination**: the test set is public, so it is in the training data, and the ranking measures exposure rather than capability. Hugging Face's Open LLM Leaderboard was eventually archived rather than patched, which is the clearest statement anyone has made about how far both had gone.\n\nSo the question is not which leaderboard is best, but which design properties resist those two failures. There are four.\n\n**A private or rotating test set.** Items that cannot be trained on because they were not published, or were published after the model was trained.\n\n**Execution-based verification.** Code that must compile and pass tests, or a task that must actually complete. Something has to work, which is far harder to fake than picking a letter.\n\n**Fresh human prompts.** People type new questions, so there is nothing to memorise, at the cost of measuring preference rather than correctness.\n\n**A published harness.** If you can rerun it yourself on a model you understand, you can calibrate the numbers.\n\n## The ones worth reading, and what each measures\n\n**Pairwise human preference arenas.** Two anonymous answers, a person picks one, ratings computed with a Bradley-Terry or Elo-style model. Resistant to contamination because the prompts are new. Measures *preference*, which rewards length, headers, confident tone and agreement with the asker, which is why the style-controlled variant of the ranking is the one to read. Note also that models are often tested under codenames, and only results a lab likes get claimed publicly.\n\n**Rotating-question benchmarks.** Evaluations that add new items monthly and score models only on items published after their training cut-off. This directly targets contamination, and it is the most useful recent development in public evaluation.\n\n**Execution-based code and agent benchmarks.** Tasks against real repositories where a patch must make failing tests pass. Expensive to run, hard to fake, and the closest public proxy for whether a model can do work.\n\n**Domain leaderboards**: embeddings, retrieval, speech recognition, translation, long context. Usually narrower and often more honest, because the communities are smaller and the metrics are older and better understood. Each has its own contamination story worth reading before quoting.\n\n**Price and speed trackers.** Independent measurement of tokens per second, time to first token and price across providers. These measure a real and stable thing, and they answer a different question from quality, which, given the earlier lesson on providers serving the same model differently, is a question you actually need answered.\n\n## Reading a ranking properly\n\n**Look for the interval.** Preference ratings come with confidence intervals, and adjacent models routinely overlap. A 15-point gap with overlapping intervals is not a ranking, it is a tie displayed in an order.\n\n**Check the sample size** behind each entry. A model with 800 votes and one with 40,000 are not measured to the same precision.\n\n**Check the date.** A three-month-old leaderboard in this field describes a different landscape.\n\n**Check what was actually run.** Which quantisation, which context length, which provider: the same warnings from the providers lesson apply to leaderboard entries served through an API.\n\n## What no leaderboard can do\n\nNone of them evaluates your task, your documents, your languages, your latency budget or your licence constraint. The correct use of a leaderboard is **shortlisting**: it narrows thirty candidates to three that are plausibly in the right class. The choice between those three is made on forty of your own examples, and that measurement outranks every public number.\n\n> The one-sentence test for whether a leaderboard deserves your attention: can you tell, from its documentation, exactly what was run against what, and could you reproduce one row of it? If not, it is entertainment."},"check":{"question":"Model A sits 12 rating points above model B on a human preference arena, with confidence intervals of plus or minus 9 points on each. What can you conclude?","options":[{"text":"Model A is better, since the point estimate is higher and intervals are conservative.","correct":false,"tempting":"Treats overlapping intervals as if the point estimates settled the question, which is what the intervals exist to prevent."},{"text":"The intervals overlap substantially, so the ordering is not established, and in any case preference is not correctness.","correct":true},{"text":"Model A is better on style but worse on substance, which is what preference rankings measure.","correct":false,"tempting":"Preference is influenced by style, but the ranking does not decompose into style and substance, and the gap is not established at all."},{"text":"More votes would resolve it, so the ranking is correct but imprecise.","correct":false,"tempting":"More votes would narrow the intervals, but until they do the ordering is unresolved rather than correct-but-imprecise."}]},"prev":{"slug":"reading-a-model-card-critically","title":"Reading a model card for what it does not say","url":"https://addaly.com/learn/open-models/reading-a-model-card-critically"},"next":{"slug":"running-the-eval-harness","title":"Running the standard harness yourself","url":"https://addaly.com/learn/open-models/running-the-eval-harness"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The leaderboards that still carry information', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/leaderboards-that-still-work (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The leaderboards that still carry information","courseTitle":"The Open Model Ecosystem","module":{"index":7,"slug":"judging-models-and-claims","title":"Judging models and claims"},"answer":null,"keyIdea":"Leaderboards die from saturation and contamination, so the ones that still carry information use private or rotating items, execution-based verification or fresh human prompts, and even then their proper use is shortlisting, not deciding.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-kills-a-leaderboard","runs":[{"text":"What kills a leaderboard"}]},{"t":"p","runs":[{"text":"Two things, and both are predictable. "},{"text":"Saturation","b":true},{"text":": everybody scores above 85 and the remaining spread is the benchmark’s own error rate. "},{"text":"Contamination","b":true},{"text":": the test set is public, so it is in the training data, and the ranking measures exposure rather than capability. Hugging Face’s Open LLM Leaderboard was eventually archived rather than patched, which is the clearest statement anyone has made about how far both had gone."}]},{"t":"p","runs":[{"text":"So the question is not which leaderboard is best, but which design properties resist those two failures. There are four."}]},{"t":"p","runs":[{"text":"A private or rotating test set.","b":true},{"text":" Items that cannot be trained on because they were not published, or were published after the model was trained."}]},{"t":"p","runs":[{"text":"Execution-based verification.","b":true},{"text":" Code that must compile and pass tests, or a task that must actually complete. Something has to work, which is far harder to fake than picking a letter."}]},{"t":"p","runs":[{"text":"Fresh human prompts.","b":true},{"text":" People type new questions, so there is nothing to memorise, at the cost of measuring preference rather than correctness."}]},{"t":"p","runs":[{"text":"A published harness.","b":true},{"text":" If you can rerun it yourself on a model you understand, you can calibrate the numbers."}]},{"t":"h2","id":"the-ones-worth-reading-and-what-each-measures","runs":[{"text":"The ones worth reading, and what each measures"}]},{"t":"p","runs":[{"text":"Pairwise human preference arenas.","b":true},{"text":" Two anonymous answers, a person picks one, ratings computed with a Bradley-Terry or Elo-style model. Resistant to contamination because the prompts are new. Measures "},{"text":"preference","i":true},{"text":", which rewards length, headers, confident tone and agreement with the asker, which is why the style-controlled variant of the ranking is the one to read. Note also that models are often tested under codenames, and only results a lab likes get claimed publicly."}]},{"t":"p","runs":[{"text":"Rotating-question benchmarks.","b":true},{"text":" Evaluations that add new items monthly and score models only on items published after their training cut-off. This directly targets contamination, and it is the most useful recent development in public evaluation."}]},{"t":"p","runs":[{"text":"Execution-based code and agent benchmarks.","b":true},{"text":" Tasks against real repositories where a patch must make failing tests pass. Expensive to run, hard to fake, and the closest public proxy for whether a model can do work."}]},{"t":"p","runs":[{"text":"Domain leaderboards","b":true},{"text":": embeddings, retrieval, speech recognition, translation, long context. Usually narrower and often more honest, because the communities are smaller and the metrics are older and better understood. Each has its own contamination story worth reading before quoting."}]},{"t":"p","runs":[{"text":"Price and speed trackers.","b":true},{"text":" Independent measurement of tokens per second, time to first token and price across providers. These measure a real and stable thing, and they answer a different question from quality, which, given the earlier lesson on providers serving the same model differently, is a question you actually need answered."}]},{"t":"h2","id":"reading-a-ranking-properly","runs":[{"text":"Reading a ranking properly"}]},{"t":"p","runs":[{"text":"Look for the interval.","b":true},{"text":" Preference ratings come with confidence intervals, and adjacent models routinely overlap. A 15-point gap with overlapping intervals is not a ranking, it is a tie displayed in an order."}]},{"t":"p","runs":[{"text":"Check the sample size","b":true},{"text":" behind each entry. A model with 800 votes and one with 40,000 are not measured to the same precision."}]},{"t":"p","runs":[{"text":"Check the date.","b":true},{"text":" A three-month-old leaderboard in this field describes a different landscape."}]},{"t":"p","runs":[{"text":"Check what was actually run.","b":true},{"text":" Which quantisation, which context length, which provider: the same warnings from the providers lesson apply to leaderboard entries served through an API."}]},{"t":"h2","id":"what-no-leaderboard-can-do","runs":[{"text":"What no leaderboard can do"}]},{"t":"p","runs":[{"text":"None of them evaluates your task, your documents, your languages, your latency budget or your licence constraint. The correct use of a leaderboard is "},{"text":"shortlisting","b":true},{"text":": it narrows thirty candidates to three that are plausibly in the right class. The choice between those three is made on forty of your own examples, and that measurement outranks every public number."}]},{"t":"quote","runs":[{"text":"The one-sentence test for whether a leaderboard deserves your attention: can you tell, from its documentation, exactly what was run against what, and could you reproduce one row of it? If not, it is entertainment."}]}],"check":{"question":[{"text":"Model A sits 12 rating points above model B on a human preference arena, with confidence intervals of plus or minus 9 points on each. What can you conclude?"}],"options":[{"text":[{"text":"Model A is better, since the point estimate is higher and intervals are conservative."}],"correct":false,"tempting":[{"text":"Treats overlapping intervals as if the point estimates settled the question, which is what the intervals exist to prevent."}]},{"text":[{"text":"The intervals overlap substantially, so the ordering is not established, and in any case preference is not correctness."}],"correct":true,"tempting":null},{"text":[{"text":"Model A is better on style but worse on substance, which is what preference rankings measure."}],"correct":false,"tempting":[{"text":"Preference is influenced by style, but the ranking does not decompose into style and substance, and the gap is not established at all."}]},{"text":[{"text":"More votes would resolve it, so the ranking is correct but imprecise."}],"correct":false,"tempting":[{"text":"More votes would narrow the intervals, but until they do the ordering is unresolved rather than correct-but-imprecise."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"reading-a-model-card-critically","title":"Reading a model card for what it does not say","minutes":8},"next":{"slug":"running-the-eval-harness","title":"Running the standard harness yourself","minutes":9}}}