Why the builder’s own numbers are weak evidence

Selection, not dishonesty

A team evaluating its own work is not usually cheating. The distortion is structural and it operates on honest people:

• You stop when the number is good. Nobody keeps digging after a result they expected and liked.
• You fix what you find, and the set stops containing it. Every failure discovered and repaired makes the remaining set easier, so the score rises for reasons that are partly real and partly bookkeeping.
• The person who wrote the prompt writes the rubric, and the rubric asks about the things the prompt was designed to do.
• The failures you never imagined are the ones your set never contains, and imagination is exactly what a builder has least of about their own design.

None of that is fraud. All of it makes the number too high in a direction nobody can quantify.

What to do inside a team

You can get much of the benefit without an external body.

A held-out set the builders never see. Owned by a different person, run by the harness, results reported as a number without item-level feedback. If the internal score is 88% and the held-out score is 74%, the gap is the size of your self-deception, and it is one of the most valuable numbers you will ever compute.

Rotate the error analysis. Whoever reads the fifty failures should not be the person who wrote the prompt, at least every second time.

Make breaking it somebody’s job. An hour a week where a colleague is explicitly rewarded for finding a failure, and where the finding is added to the set, not argued with, does more than any process document.

Have a person outside the team read the card. They will ask what “supported” means, and that question is the audit.

External evaluation, and its immaturity

Beyond the team, the options are third-party audits, structured access for researchers, and public bug-bounty-style programmes for AI harms. All are young, and it is worth being blunt about the state of things: there is no established profession here comparable to financial audit, no agreed standard for what an AI audit covers, and no licensing regime. “Independently verified” frequently means a consultancy ran the vendor’s own test suite.

So when reading someone else’s independent evaluation, ask three questions: did the evaluator choose the set, was the protocol fixed before results were seen, and was publication guaranteed regardless of outcome. If the answer to any is no, it is a review, not an audit.

Those are also the three commitments to make when you commission one.

Reading somebody else’s evaluation

The same scepticism applies when a supplier, a competitor or a research group publishes numbers about their own system. Four questions extract most of the value from a page of results:

• What was the set, and who built it? A set assembled by the party being measured is the weakest form.
• What was left out? Look for the slice that is conspicuously absent: usually a language, a difficulty band, or the adversarial case.
• Which baseline? A result with no baseline cannot be read at all, as the first block of this course established.
• Is there an interval and a sample size? A percentage without them is a claim rather than a measurement.

None of this requires access to their data. It is all visible in what they chose to print.

The trade you cannot avoid

A private evaluation set is uncontaminated and unverifiable. Nobody can check your claim, and you can quietly change the set.

A public evaluation set is verifiable and decays. Once published, it enters training corpora, and within a year a high score may mean the model has seen the answers. This is not hypothetical: it is the documented life cycle of every major public benchmark.

There is no way to have both, and the practical compromise is a split:

• Publish the methodology in full, plus a sample of items, so a reader can judge the design and reproduce the procedure on their own data.
• Hold the rest privately, and rotate a portion of it each year so that even leakage through use has a limited life.
• Say which parts are which in the report, so a reader knows what they are relying on.

That is weaker than an audited number and stronger than a claim, and it is the honest position available today.