A score is not a property of a model
A benchmark number is produced by a pipeline: the model, plus the prompt format, plus the number of examples shown, plus the decoding settings, plus the regex that pulls an answer out of the text, plus the harness that glues it together. Change any one and the number moves, often by several points.
This is why two reports of "MMLU" for the same model can differ. Nobody lied. They ran different evaluations with the same name. When you see a comparison, the first question is not "how big is the gap" but "was this the same pipeline for both".
Contamination
Benchmark test sets live on the public web. Models are trained on the public web. Some of the test set ends up in training, and the model recites rather than solves.
The cleanest evidence is GSM1K, published in 2024: researchers built a fresh set of grade-school maths problems matched in style and difficulty to GSM8K, and re-tested. Some model families dropped by up to around thirteen points. Others were flat. The size of the drop is a decent estimate of how much of the original score was memory.
Most contamination is accidental. Nobody has to cheat for a widely mirrored test set to land in a web crawl. But accidental contamination inflates the number just as much as deliberate contamination does.
Gaming, in ascending order of dishonesty
- Picking the favourable configuration. Trying several prompt formats and shot counts and reporting the best. Nearly universal, rarely disclosed.
- Undisclosed aggregation. Reporting majority vote at k, or best-of-n, next to someone else's single-sample number. A maj@64 score is not a score you will ever see in production.
- Training on benchmark-shaped data. Adding a large pile of grade-school word problems is a legitimate training choice that also destroys GSM8K's ability to predict anything else.
- Training on the test set. Rare among reputable labs. Not unknown.
Arenas measure preference, which is a different thing
Human preference leaderboards are harder to contaminate, because the prompts are fresh. They measure what people pick when shown two answers, which rewards length, headers, bullet lists, confident tone, and agreement with the asker. Style-controlled variants of these rankings exist and are the ones worth reading. Note also that models are often tested under anonymous codenames, and only the results a lab likes get claimed.
The benchmark itself has a noise floor
An audit of a 3,000-question sample of MMLU found problems with roughly 6.5% of the items — ambiguous questions, wrong keys, unanswerable prompts. Cleaned and harder variants exist for that reason. When two models are within two or three points, a meaningful share of the difference is the benchmark's own errors. Hugging Face's Open LLM Leaderboard was eventually archived rather than patched, which tells you how far saturation and contamination had gone.
What still carries information
- Held-out or refreshed sets. Evaluations that rotate questions monthly, or keep a private split, cannot be trained on in advance.
- Benchmarks with an execution harness. Code that must compile and pass tests, or agent tasks that must actually complete. Something has to work, which is much harder to fake than a multiple-choice letter.
- Papers that publish ablations and failures. A lab showing where its model is worse is giving you calibration on the numbers where it is better.
- Twenty outputs on your own inputs. This beats every leaderboard for your decision, and the last lesson is about doing it.
Six questions for any launch post
Which harness. How many shots. Single sample or aggregated. Who ran the baseline numbers — them, or copied from another paper. Is the gap larger than three points. Does the benchmark look anything like your task.
If you cannot answer four of those, the post is marketing. That does not mean the model is bad. It means the post is not evidence.
Before you move on