{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"a-judge-you-have-checked","title":"A model judge you have checked against people","url":"https://addaly.com/learn/building-with-ai/a-judge-you-have-checked","position":55,"of":80,"minutes":9,"answer":"A model can grade your feature's answers at scale only once you have measured its agreement with human labels on a sample, written the rubric as binary criteria rather than a score.","keyIdea":"A model can grade your feature's answers at scale only once you have measured its agreement with human labels on a sample, written the rubric as binary criteria rather than a score, swapped the order to expose position bias, and used a different model from the one being judged."},"body":{"format":"markdown","text":"## The layer above assertions\n\nAn answer passes every mechanical check and is still wrong. Someone has to read it. At fifty cases that someone can be you; at five hundred, or at every commit, it has to be a model. Using a model to grade a model is standard practice now, and it works: under conditions that most teams skip. The evaluation course devotes a block to the biases of judges; this lesson is what you must do before trusting one on a feature you are shipping.\n\n## Write the rubric as questions with yes-or-no answers\n\nA judge asked for a score out of ten returns numbers that drift with phrasing, cluster around seven, and mean nothing across prompt versions. A judge asked binary questions returns verdicts you can count.\n\n```text\nFor the answer below, reply with JSON: {\"correct_window\": bool,\n\"applies_from_purchase\": bool, \"no_invented_policy\": bool,\n\"reason\": \"one sentence\"}\ncorrect_window: does the answer state the refund window is 14 days?\napplies_from_purchase: does it say the window runs from purchase, not renewal?\nno_invented_policy: does it avoid stating any policy not in the reference?\n```\n\nEach criterion is a fact about the answer. The case passes if all required criteria are true. The `reason` field is not decoration: it is what you read when the verdict looks wrong, and it is how you find rubric criteria the judge is misreading.\n\n## Pairwise when there is no reference\n\nFor open-ended output (a draft email, a summary) there is often no reference answer, and \"is this good?\" is unanswerable. \"Is A better than B?\" is answerable. Show the judge two outputs, from the old prompt and the new, and ask which better satisfies the rubric. A comparison is easier and more consistent than an absolute grade, and it is what you actually want to know when deciding whether to ship a change.\n\n## Position bias, and the swap\n\nJudges prefer the first option shown, or the second, by a margin that varies by model and is never zero. The test is simple: run every pair both ways. If the judge prefers A in the A-then-B order and B in the B-then-A order, the verdict is position, not quality, and that pair is a tie. Report the tie rate; if it is high, the two prompts are not different enough to distinguish, which is a finding.\n\n## Self-preference\n\nA model judging its own outputs favours them: the same style, the same phrasing choices read as correct to it. Judge with a different model family from the one generating, or with a stronger model from the same family if that is all you have, and say which in the report.\n\n## The calibration step, which is the whole point\n\nBefore the judge grades anything that decides a release, grade fifty answers by hand (two people, independently, then reconcile) and run the judge on the same fifty. Compare. Simple agreement, the share of answers where judge and humans reached the same verdict, is the number to write down; the evaluation course covers the chance-corrected version. Above ninety per cent, use the judge and quote the figure. Between eighty and ninety, use it for triage but not for decisions. Below eighty, the rubric is unclear or the task is beyond the judge, and every number it produces afterwards is noise with a decimal point.\n\nRedo this whenever the rubric, the judge model or the feature changes materially. Agreement measured in March against a judge model that was retired in June is not a figure you can quote in July.\n\n## Cost per run\n\nA judge call per case, with the rubric, the reference and the answer, is a few thousand tokens. Fifty cases is a few hundred thousand tokens: well under a dollar with a mid-sized model, a few dollars with a frontier one. Pairwise doubles it; the order swap doubles it again. Cheap enough to run on every prompt change; expensive enough that the assertion layer should filter first, so the judge only reads answers that were not mechanically broken.\n\n## What a judge cannot know\n\nA judge sees the answer and whatever reference you gave it. It does not know whether the reference is right, whether the corpus has changed, or whether the user's real question was different from the one typed. It grades against the rubric, and a rubric that encodes a mistaken policy will pass answers that repeat the mistake. That is why the error-analysis lesson later in this block keeps \"expectation wrong\" as a category, and why a judge's verdict is evidence to be examined rather than a decision made.\n\n## Reading the disagreements\n\nThe calibration sample's disagreements are the most useful artefact this lesson produces. Each one is either a rubric criterion the judge misreads, a case whose expectation is wrong, or a genuine hard call. Sorting the fifty into those three piles takes an hour and improves the rubric more than any amount of prompt engineering on the judge itself."},"check":{"question":"A judge model prefers the new prompt's answer in 70 per cent of pairs when it is shown first, and the old prompt's answer in 65 per cent of pairs when that is shown first. What should the team conclude?","options":[{"text":"The new prompt is better, since 70 exceeds 65.","correct":false,"tempting":"Compares two numbers that each measure position preference, not quality; the same pairs reversed their verdict."},{"text":"The judge is broken and should be replaced with a stronger model before any further use, since a reliable judge would not reverse its own verdicts.","correct":false,"tempting":"Every judge shows some position bias; the remedy is the swap and tie-counting, not replacement."},{"text":"Position is driving the verdicts, so most pairs are ties and this judge cannot distinguish the two prompts.","correct":true},{"text":"The rubric needs to be rewritten as a score out of ten so the judge can express smaller differences.","correct":false,"tempting":"Numeric scores add drift without removing position bias, and binary criteria are the more reliable form."}]},"prev":{"slug":"assertions-before-judges","title":"Assertions before judges: the checks that cost nothing","url":"https://addaly.com/learn/building-with-ai/assertions-before-judges"},"next":{"slug":"measuring-retrieval-on-its-own","title":"Measuring retrieval on its own","url":"https://addaly.com/learn/building-with-ai/measuring-retrieval-on-its-own"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'A model judge you have checked against people', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/a-judge-you-have-checked (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"A model judge you have checked against people","courseTitle":"Building with AI","module":{"index":6,"slug":"evaluating-and-iterating","title":"Evaluating and iterating"},"answer":null,"keyIdea":"A model can grade your feature’s answers at scale only once you have measured its agreement with human labels on a sample, written the rubric as binary criteria rather than a score, swapped the order to expose position bias, and used a different model from the one being judged.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-layer-above-assertions","runs":[{"text":"The layer above assertions"}]},{"t":"p","runs":[{"text":"An answer passes every mechanical check and is still wrong. Someone has to read it. At fifty cases that someone can be you; at five hundred, or at every commit, it has to be a model. Using a model to grade a model is standard practice now, and it works: under conditions that most teams skip. The evaluation course devotes a block to the biases of judges; this lesson is what you must do before trusting one on a feature you are shipping."}]},{"t":"h2","id":"write-the-rubric-as-questions-with-yes-or-no-answers","runs":[{"text":"Write the rubric as questions with yes-or-no answers"}]},{"t":"p","runs":[{"text":"A judge asked for a score out of ten returns numbers that drift with phrasing, cluster around seven, and mean nothing across prompt versions. A judge asked binary questions returns verdicts you can count."}]},{"t":"code","lang":"text","label":"Text","text":"For the answer below, reply with JSON: {\"correct_window\": bool,\n\"applies_from_purchase\": bool, \"no_invented_policy\": bool,\n\"reason\": \"one sentence\"}\ncorrect_window: does the answer state the refund window is 14 days?\napplies_from_purchase: does it say the window runs from purchase, not renewal?\nno_invented_policy: does it avoid stating any policy not in the reference?"},{"t":"p","runs":[{"text":"Each criterion is a fact about the answer. The case passes if all required criteria are true. The "},{"text":"reason","code":true},{"text":" field is not decoration: it is what you read when the verdict looks wrong, and it is how you find rubric criteria the judge is misreading."}]},{"t":"h2","id":"pairwise-when-there-is-no-reference","runs":[{"text":"Pairwise when there is no reference"}]},{"t":"p","runs":[{"text":"For open-ended output (a draft email, a summary) there is often no reference answer, and “is this good?” is unanswerable. “Is A better than B?” is answerable. Show the judge two outputs, from the old prompt and the new, and ask which better satisfies the rubric. A comparison is easier and more consistent than an absolute grade, and it is what you actually want to know when deciding whether to ship a change."}]},{"t":"h2","id":"position-bias-and-the-swap","runs":[{"text":"Position bias, and the swap"}]},{"t":"p","runs":[{"text":"Judges prefer the first option shown, or the second, by a margin that varies by model and is never zero. The test is simple: run every pair both ways. If the judge prefers A in the A-then-B order and B in the B-then-A order, the verdict is position, not quality, and that pair is a tie. Report the tie rate; if it is high, the two prompts are not different enough to distinguish, which is a finding."}]},{"t":"h2","id":"self-preference","runs":[{"text":"Self-preference"}]},{"t":"p","runs":[{"text":"A model judging its own outputs favours them: the same style, the same phrasing choices read as correct to it. Judge with a different model family from the one generating, or with a stronger model from the same family if that is all you have, and say which in the report."}]},{"t":"h2","id":"the-calibration-step-which-is-the-whole-point","runs":[{"text":"The calibration step, which is the whole point"}]},{"t":"p","runs":[{"text":"Before the judge grades anything that decides a release, grade fifty answers by hand (two people, independently, then reconcile) and run the judge on the same fifty. Compare. Simple agreement, the share of answers where judge and humans reached the same verdict, is the number to write down; the evaluation course covers the chance-corrected version. Above ninety per cent, use the judge and quote the figure. Between eighty and ninety, use it for triage but not for decisions. Below eighty, the rubric is unclear or the task is beyond the judge, and every number it produces afterwards is noise with a decimal point."}]},{"t":"p","runs":[{"text":"Redo this whenever the rubric, the judge model or the feature changes materially. Agreement measured in March against a judge model that was retired in June is not a figure you can quote in July."}]},{"t":"h2","id":"cost-per-run","runs":[{"text":"Cost per run"}]},{"t":"p","runs":[{"text":"A judge call per case, with the rubric, the reference and the answer, is a few thousand tokens. Fifty cases is a few hundred thousand tokens: well under a dollar with a mid-sized model, a few dollars with a frontier one. Pairwise doubles it; the order swap doubles it again. Cheap enough to run on every prompt change; expensive enough that the assertion layer should filter first, so the judge only reads answers that were not mechanically broken."}]},{"t":"h2","id":"what-a-judge-cannot-know","runs":[{"text":"What a judge cannot know"}]},{"t":"p","runs":[{"text":"A judge sees the answer and whatever reference you gave it. It does not know whether the reference is right, whether the corpus has changed, or whether the user’s real question was different from the one typed. It grades against the rubric, and a rubric that encodes a mistaken policy will pass answers that repeat the mistake. That is why the error-analysis lesson later in this block keeps “expectation wrong” as a category, and why a judge’s verdict is evidence to be examined rather than a decision made."}]},{"t":"h2","id":"reading-the-disagreements","runs":[{"text":"Reading the disagreements"}]},{"t":"p","runs":[{"text":"The calibration sample’s disagreements are the most useful artefact this lesson produces. Each one is either a rubric criterion the judge misreads, a case whose expectation is wrong, or a genuine hard call. Sorting the fifty into those three piles takes an hour and improves the rubric more than any amount of prompt engineering on the judge itself."}]}],"check":{"question":[{"text":"A judge model prefers the new prompt’s answer in 70 per cent of pairs when it is shown first, and the old prompt’s answer in 65 per cent of pairs when that is shown first. What should the team conclude?"}],"options":[{"text":[{"text":"The new prompt is better, since 70 exceeds 65."}],"correct":false,"tempting":[{"text":"Compares two numbers that each measure position preference, not quality; the same pairs reversed their verdict."}]},{"text":[{"text":"The judge is broken and should be replaced with a stronger model before any further use, since a reliable judge would not reverse its own verdicts."}],"correct":false,"tempting":[{"text":"Every judge shows some position bias; the remedy is the swap and tie-counting, not replacement."}]},{"text":[{"text":"Position is driving the verdicts, so most pairs are ties and this judge cannot distinguish the two prompts."}],"correct":true,"tempting":null},{"text":[{"text":"The rubric needs to be rewritten as a score out of ten so the judge can express smaller differences."}],"correct":false,"tempting":[{"text":"Numeric scores add drift without removing position bias, and binary criteria are the more reliable form."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"assertions-before-judges","title":"Assertions before judges: the checks that cost nothing","minutes":8},"next":{"slug":"measuring-retrieval-on-its-own","title":"Measuring retrieval on its own","minutes":8}}}