{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"independent-evaluation","title":"Why the builder's own numbers are weak evidence","url":"https://addaly.com/learn/evaluating-ai/independent-evaluation","position":61,"of":76,"minutes":8,"answer":"The team that built a system chooses the set, the metric and what gets reported, so arrange for someone else to hold a set you never see.","keyIdea":"The team that built a system chooses the set, the metric and what gets reported, so arrange for someone else to hold a set you never see, and accept that publishing an evaluation set is what destroys it."},"body":{"format":"markdown","text":"## Selection, not dishonesty\n\nA team evaluating its own work is not usually cheating. The distortion is structural and it operates on honest people:\n\n- **You stop when the number is good.** Nobody keeps digging after a result they expected and liked.\n- **You fix what you find, and the set stops containing it.** Every failure discovered and repaired makes the remaining set easier, so the score rises for reasons that are partly real and partly bookkeeping.\n- **The person who wrote the prompt writes the rubric**, and the rubric asks about the things the prompt was designed to do.\n- **The failures you never imagined are the ones your set never contains**, and imagination is exactly what a builder has least of about their own design.\n\nNone of that is fraud. All of it makes the number too high in a direction nobody can quantify.\n\n## What to do inside a team\n\nYou can get much of the benefit without an external body.\n\n**A held-out set the builders never see.** Owned by a different person, run by the harness, results reported as a number without item-level feedback. If the internal score is 88% and the held-out score is 74%, the gap is the size of your self-deception, and it is one of the most valuable numbers you will ever compute.\n\n**Rotate the error analysis.** Whoever reads the fifty failures should not be the person who wrote the prompt, at least every second time.\n\n**Make breaking it somebody's job.** An hour a week where a colleague is explicitly rewarded for finding a failure, and where the finding is added to the set, not argued with, does more than any process document.\n\n**Have a person outside the team read the card.** They will ask what \"supported\" means, and that question is the audit.\n\n## External evaluation, and its immaturity\n\nBeyond the team, the options are third-party audits, structured access for researchers, and public bug-bounty-style programmes for AI harms. All are young, and it is worth being blunt about the state of things: there is no established profession here comparable to financial audit, no agreed standard for what an AI audit covers, and no licensing regime. \"Independently verified\" frequently means a consultancy ran the vendor's own test suite.\n\nSo when reading someone else's independent evaluation, ask three questions: did the evaluator choose the set, was the protocol fixed before results were seen, and was publication guaranteed regardless of outcome. If the answer to any is no, it is a review, not an audit.\n\nThose are also the three commitments to make when you commission one.\n\n## Reading somebody else's evaluation\n\nThe same scepticism applies when a supplier, a competitor or a research group publishes numbers about their own system. Four questions extract most of the value from a page of results:\n\n- **What was the set, and who built it?** A set assembled by the party being measured is the weakest form.\n- **What was left out?** Look for the slice that is conspicuously absent: usually a language, a difficulty band, or the adversarial case.\n- **Which baseline?** A result with no baseline cannot be read at all, as the first block of this course established.\n- **Is there an interval and a sample size?** A percentage without them is a claim rather than a measurement.\n\nNone of this requires access to their data. It is all visible in what they chose to print.\n\n## The trade you cannot avoid\n\nA private evaluation set is uncontaminated and unverifiable. Nobody can check your claim, and you can quietly change the set.\n\nA public evaluation set is verifiable and decays. Once published, it enters training corpora, and within a year a high score may mean the model has seen the answers. This is not hypothetical: it is the documented life cycle of every major public benchmark.\n\nThere is no way to have both, and the practical compromise is a split:\n\n- **Publish the methodology in full**, plus a sample of items, so a reader can judge the design and reproduce the procedure on their own data.\n- **Hold the rest privately**, and rotate a portion of it each year so that even leakage through use has a limited life.\n- **Say which parts are which** in the report, so a reader knows what they are relying on.\n\nThat is weaker than an audited number and stronger than a claim, and it is the honest position available today."},"check":{"question":"A team's internal evaluation reports 88% while a set held by another team, which the builders never see, reports 74%. What is the most useful reading?","options":[{"text":"The held-out set is harder, so the two numbers are not comparable and only the trend within each matters.","correct":false,"tempting":"Possible and untested; treating the gap as a set-difficulty artefact removes the one diagnostic the arrangement exists to give you."},{"text":"The gap estimates how much the internal number was inflated by building and fixing against that set, and it is the number worth reporting outside the team.","correct":true},{"text":"The internal set has been contaminated and should be discarded and rebuilt.","correct":false,"tempting":"Overreacts: the internal set remains useful for development, and its role is to guide work rather than to produce the reported figure."},{"text":"The held-out evaluation was run by people less familiar with the task, so its labels are probably noisier.","correct":false,"tempting":"Label quality is worth checking, and it does not account for a 14-point gap in the direction that a builder's self-selection predicts."}]},"prev":{"slug":"evaluating-a-vendor-model","title":"Buying a model you will depend on","url":"https://addaly.com/learn/evaluating-ai/evaluating-a-vendor-model"},"next":{"slug":"standards-and-frameworks","title":"What the emerging rules ask for, and what they do not settle","url":"https://addaly.com/learn/evaluating-ai/standards-and-frameworks"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Why the builder's own numbers are weak evidence', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/independent-evaluation (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Why the builder’s own numbers are weak evidence","courseTitle":"Knowing If It Works","module":{"index":7,"slug":"assurance","title":"Evidence other people can act on"},"answer":null,"keyIdea":"The team that built a system chooses the set, the metric and what gets reported, so arrange for someone else to hold a set you never see, and accept that publishing an evaluation set is what destroys it.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"selection-not-dishonesty","runs":[{"text":"Selection, not dishonesty"}]},{"t":"p","runs":[{"text":"A team evaluating its own work is not usually cheating. The distortion is structural and it operates on honest people:"}]},{"t":"ul","items":[[{"text":"You stop when the number is good.","b":true},{"text":" Nobody keeps digging after a result they expected and liked."}],[{"text":"You fix what you find, and the set stops containing it.","b":true},{"text":" Every failure discovered and repaired makes the remaining set easier, so the score rises for reasons that are partly real and partly bookkeeping."}],[{"text":"The person who wrote the prompt writes the rubric","b":true},{"text":", and the rubric asks about the things the prompt was designed to do."}],[{"text":"The failures you never imagined are the ones your set never contains","b":true},{"text":", and imagination is exactly what a builder has least of about their own design."}]]},{"t":"p","runs":[{"text":"None of that is fraud. All of it makes the number too high in a direction nobody can quantify."}]},{"t":"h2","id":"what-to-do-inside-a-team","runs":[{"text":"What to do inside a team"}]},{"t":"p","runs":[{"text":"You can get much of the benefit without an external body."}]},{"t":"p","runs":[{"text":"A held-out set the builders never see.","b":true},{"text":" Owned by a different person, run by the harness, results reported as a number without item-level feedback. If the internal score is 88% and the held-out score is 74%, the gap is the size of your self-deception, and it is one of the most valuable numbers you will ever compute."}]},{"t":"p","runs":[{"text":"Rotate the error analysis.","b":true},{"text":" Whoever reads the fifty failures should not be the person who wrote the prompt, at least every second time."}]},{"t":"p","runs":[{"text":"Make breaking it somebody’s job.","b":true},{"text":" An hour a week where a colleague is explicitly rewarded for finding a failure, and where the finding is added to the set, not argued with, does more than any process document."}]},{"t":"p","runs":[{"text":"Have a person outside the team read the card.","b":true},{"text":" They will ask what “supported” means, and that question is the audit."}]},{"t":"h2","id":"external-evaluation-and-its-immaturity","runs":[{"text":"External evaluation, and its immaturity"}]},{"t":"p","runs":[{"text":"Beyond the team, the options are third-party audits, structured access for researchers, and public bug-bounty-style programmes for AI harms. All are young, and it is worth being blunt about the state of things: there is no established profession here comparable to financial audit, no agreed standard for what an AI audit covers, and no licensing regime. “Independently verified” frequently means a consultancy ran the vendor’s own test suite."}]},{"t":"p","runs":[{"text":"So when reading someone else’s independent evaluation, ask three questions: did the evaluator choose the set, was the protocol fixed before results were seen, and was publication guaranteed regardless of outcome. If the answer to any is no, it is a review, not an audit."}]},{"t":"p","runs":[{"text":"Those are also the three commitments to make when you commission one."}]},{"t":"h2","id":"reading-somebody-else-s-evaluation","runs":[{"text":"Reading somebody else’s evaluation"}]},{"t":"p","runs":[{"text":"The same scepticism applies when a supplier, a competitor or a research group publishes numbers about their own system. Four questions extract most of the value from a page of results:"}]},{"t":"ul","items":[[{"text":"What was the set, and who built it?","b":true},{"text":" A set assembled by the party being measured is the weakest form."}],[{"text":"What was left out?","b":true},{"text":" Look for the slice that is conspicuously absent: usually a language, a difficulty band, or the adversarial case."}],[{"text":"Which baseline?","b":true},{"text":" A result with no baseline cannot be read at all, as the first block of this course established."}],[{"text":"Is there an interval and a sample size?","b":true},{"text":" A percentage without them is a claim rather than a measurement."}]]},{"t":"p","runs":[{"text":"None of this requires access to their data. It is all visible in what they chose to print."}]},{"t":"h2","id":"the-trade-you-cannot-avoid","runs":[{"text":"The trade you cannot avoid"}]},{"t":"p","runs":[{"text":"A private evaluation set is uncontaminated and unverifiable. Nobody can check your claim, and you can quietly change the set."}]},{"t":"p","runs":[{"text":"A public evaluation set is verifiable and decays. Once published, it enters training corpora, and within a year a high score may mean the model has seen the answers. This is not hypothetical: it is the documented life cycle of every major public benchmark."}]},{"t":"p","runs":[{"text":"There is no way to have both, and the practical compromise is a split:"}]},{"t":"ul","items":[[{"text":"Publish the methodology in full","b":true},{"text":", plus a sample of items, so a reader can judge the design and reproduce the procedure on their own data."}],[{"text":"Hold the rest privately","b":true},{"text":", and rotate a portion of it each year so that even leakage through use has a limited life."}],[{"text":"Say which parts are which","b":true},{"text":" in the report, so a reader knows what they are relying on."}]]},{"t":"p","runs":[{"text":"That is weaker than an audited number and stronger than a claim, and it is the honest position available today."}]}],"check":{"question":[{"text":"A team’s internal evaluation reports 88% while a set held by another team, which the builders never see, reports 74%. What is the most useful reading?"}],"options":[{"text":[{"text":"The held-out set is harder, so the two numbers are not comparable and only the trend within each matters."}],"correct":false,"tempting":[{"text":"Possible and untested; treating the gap as a set-difficulty artefact removes the one diagnostic the arrangement exists to give you."}]},{"text":[{"text":"The gap estimates how much the internal number was inflated by building and fixing against that set, and it is the number worth reporting outside the team."}],"correct":true,"tempting":null},{"text":[{"text":"The internal set has been contaminated and should be discarded and rebuilt."}],"correct":false,"tempting":[{"text":"Overreacts: the internal set remains useful for development, and its role is to guide work rather than to produce the reported figure."}]},{"text":[{"text":"The held-out evaluation was run by people less familiar with the task, so its labels are probably noisier."}],"correct":false,"tempting":[{"text":"Label quality is worth checking, and it does not account for a 14-point gap in the direction that a builder’s self-selection predicts."}]}],"order":[2,1,3,0],"why":null},"prev":{"slug":"evaluating-a-vendor-model","title":"Buying a model you will depend on","minutes":10},"next":{"slug":"standards-and-frameworks","title":"What the emerging rules ask for, and what they do not settle","minutes":9}}}