{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"calibrating-and-monitoring-the-judge","title":"Proving the judge agrees with you, and re-proving it","url":"https://addaly.com/learn/evaluating-ai/calibrating-and-monitoring-the-judge","position":43,"of":76,"minutes":9,"answer":"Judge quality is per-class recall on a stratified calibration set, not overall agreement, and it must be re-measured whenever the judge model, the prompt or the traffic changes.","keyIdea":"Judge quality is per-class recall on a stratified calibration set, not overall agreement, and it must be re-measured whenever the judge model, the prompt or the traffic changes."},"body":{"format":"markdown","text":"## The calibration set\n\nA judge is a measuring instrument, and an uncalibrated instrument produces numbers, not measurements. Calibration here means one specific thing: showing that the judge's verdicts match careful human verdicts on the same items, and stating by how much they differ.\n\nBuild a **calibration set** and keep it separate from everything else:\n\n- 50 to 100 items, drawn from real traffic\n- labelled independently by two people, disagreements resolved and the ruling written down\n- **deliberately enriched with the rare class.** If violations are 4% of traffic, a random 50 contains two of them and tells you nothing about whether the judge catches violations. Take 25 known violations and 25 clean items, and remember when you report that you have stratified.\n- frozen, versioned, and never used for tuning the system itself\n\nThat set costs about five hours of two people's time once, and it is what every judge number you ever quote will rest on.\n\n## Agreement, per class\n\nRun the judge over the calibration set and compare against the human labels. Compute overall agreement; and then immediately stop looking at it and compute the two numbers that matter.\n\nAn illustrative result on a set of 50 with 12 real violations:\n\n- Overall agreement: 44/50, which is 88%. Sounds fine.\n- Of the 12 real violations, the judge flagged 4. **Recall 33%.**\n- Of the 6 items it flagged, 4 were real. **Precision 67%.**\n\nThe judge is 88% agreeable and catches one violation in three. If you had shipped a dashboard on the overall figure, you would have reported a low violation rate and been technically correct and completely wrong.\n\nThis is the same lesson as accuracy under imbalance from module two, and it bites harder here because the number feels like a validation rather than a metric.\n\nUse **Cohen's kappa** as the headline instead, for the reason given in module one: it discounts the agreement you would get by chance. Above 0.8, the judge is a usable instrument. Between 0.6 and 0.8, usable with the slack stated on every number it produces. Below 0.6, it is not measuring your criterion and no amount of prompt fiddling should be reported as if it were.\n\n## What to do when agreement is poor\n\nIn order of how often it works:\n\n1. **Split the rule.** Most poor agreement comes from a compound criterion. Two rules at 0.85 beat one at 0.55.\n2. **Add the boundary rulings** from your labelling guide directly into the judge prompt as few-shot examples. Two or three well-chosen edge cases move agreement more than any amount of instruction rewriting.\n3. **Require quoted evidence** before the verdict, if you are not already.\n4. **Change judge model.** A larger model helps on subtle criteria. It also costs more per call, so measure whether it earns it.\n5. **Accept that it is a human criterion.** Some judgements (whether a bereavement message was appropriate, whether a joke lands in Marathi) are not automatable at any agreement you would accept. Sample these for human review and stop pretending.\n\nNever accept the fifth option before trying the first.\n\n## Re-calibrate on a schedule and on a trigger\n\nA calibration goes stale. Re-run the frozen calibration set:\n\n- **monthly**, as routine\n- **whenever the judge model version changes**, including when a provider silently updates an alias like `-latest`, which is the argument for pinning an exact version string in the judge as much as in the system\n- **whenever the judge prompt changes**, even trivially\n- **whenever the input distribution moves**, such as launching in a new language or a new customer segment\n\nThis takes ten minutes because the labels already exist. Skipping it is how a team ends up six months into trusting a judge that stopped agreeing with them in March.\n\nRecord each run: date, judge model version, judge prompt version, kappa, per-class recall. Five lines in a file. That file is the provenance of every quality number your organisation believes.\n\n## The bias you cannot calibrate away\n\nOne caveat to end on. Calibration proves the judge agrees with your labels. It does not prove your labels are right. If your two labellers share an assumption (that longer answers are more thorough, that formal register is more professional) the judge will learn that assumption and the calibration will look excellent.\n\nThe only defence is periodically bringing in somebody outside the team to label 20 items cold and comparing. It is uncomfortable and it is the cheapest audit available."},"check":{"question":"A judge is validated on 200 randomly sampled production items and agrees with human labels 94% of the time. Policy violations occur in about 3% of traffic. Why is this validation weak?","options":[{"text":"A random sample of 200 contains roughly six violations, so the figure says almost nothing about whether the judge detects the thing it exists to detect.","correct":true},{"text":"200 items is too small for any reliable agreement estimate; at least 1,000 are needed.","correct":false,"tempting":"Blames overall sample size when the shortage is specifically of positive cases, which stratifying fixes without any increase in labelling effort."},{"text":"Human labels on production data are unreliable because reviewers see the model's output before deciding.","correct":false,"tempting":"Raises an anchoring risk worth controlling for, but it does not explain why a 94% figure on this sample is uninformative."},{"text":"Agreement should be measured against a second judge model rather than humans, to remove human inconsistency.","correct":false,"tempting":"Replaces the ground truth with another uncalibrated model, which measures whether two models share a bias rather than whether either is right."}]},"prev":{"slug":"reference-answers-and-equivalence","title":"Gold answers, and asking whether two answers agree","url":"https://addaly.com/learn/evaluating-ai/reference-answers-and-equivalence"},"next":{"slug":"ranking-systems-by-preference","title":"Turning pairwise wins into a ranking you can defend","url":"https://addaly.com/learn/evaluating-ai/ranking-systems-by-preference"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Proving the judge agrees with you, and re-proving it', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/calibrating-and-monitoring-the-judge (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Proving the judge agrees with you, and re-proving it","courseTitle":"Knowing If It Works","module":{"index":5,"slug":"automating-judgement","title":"Automating judgement"},"answer":null,"keyIdea":"Judge quality is per-class recall on a stratified calibration set, not overall agreement, and it must be re-measured whenever the judge model, the prompt or the traffic changes.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-calibration-set","runs":[{"text":"The calibration set"}]},{"t":"p","runs":[{"text":"A judge is a measuring instrument, and an uncalibrated instrument produces numbers, not measurements. Calibration here means one specific thing: showing that the judge’s verdicts match careful human verdicts on the same items, and stating by how much they differ."}]},{"t":"p","runs":[{"text":"Build a "},{"text":"calibration set","b":true},{"text":" and keep it separate from everything else:"}]},{"t":"ul","items":[[{"text":"50 to 100 items, drawn from real traffic"}],[{"text":"labelled independently by two people, disagreements resolved and the ruling written down"}],[{"text":"deliberately enriched with the rare class.","b":true},{"text":" If violations are 4% of traffic, a random 50 contains two of them and tells you nothing about whether the judge catches violations. Take 25 known violations and 25 clean items, and remember when you report that you have stratified."}],[{"text":"frozen, versioned, and never used for tuning the system itself"}]]},{"t":"p","runs":[{"text":"That set costs about five hours of two people’s time once, and it is what every judge number you ever quote will rest on."}]},{"t":"h2","id":"agreement-per-class","runs":[{"text":"Agreement, per class"}]},{"t":"p","runs":[{"text":"Run the judge over the calibration set and compare against the human labels. Compute overall agreement; and then immediately stop looking at it and compute the two numbers that matter."}]},{"t":"p","runs":[{"text":"An illustrative result on a set of 50 with 12 real violations:"}]},{"t":"ul","items":[[{"text":"Overall agreement: 44/50, which is 88%. Sounds fine."}],[{"text":"Of the 12 real violations, the judge flagged 4. "},{"text":"Recall 33%.","b":true}],[{"text":"Of the 6 items it flagged, 4 were real. "},{"text":"Precision 67%.","b":true}]]},{"t":"p","runs":[{"text":"The judge is 88% agreeable and catches one violation in three. If you had shipped a dashboard on the overall figure, you would have reported a low violation rate and been technically correct and completely wrong."}]},{"t":"p","runs":[{"text":"This is the same lesson as accuracy under imbalance from module two, and it bites harder here because the number feels like a validation rather than a metric."}]},{"t":"p","runs":[{"text":"Use "},{"text":"Cohen’s kappa","b":true},{"text":" as the headline instead, for the reason given in module one: it discounts the agreement you would get by chance. Above 0.8, the judge is a usable instrument. Between 0.6 and 0.8, usable with the slack stated on every number it produces. Below 0.6, it is not measuring your criterion and no amount of prompt fiddling should be reported as if it were."}]},{"t":"h2","id":"what-to-do-when-agreement-is-poor","runs":[{"text":"What to do when agreement is poor"}]},{"t":"p","runs":[{"text":"In order of how often it works:"}]},{"t":"ol","start":1,"items":[[{"text":"Split the rule.","b":true},{"text":" Most poor agreement comes from a compound criterion. Two rules at 0.85 beat one at 0.55."}],[{"text":"Add the boundary rulings","b":true},{"text":" from your labelling guide directly into the judge prompt as few-shot examples. Two or three well-chosen edge cases move agreement more than any amount of instruction rewriting."}],[{"text":"Require quoted evidence","b":true},{"text":" before the verdict, if you are not already."}],[{"text":"Change judge model.","b":true},{"text":" A larger model helps on subtle criteria. It also costs more per call, so measure whether it earns it."}],[{"text":"Accept that it is a human criterion.","b":true},{"text":" Some judgements (whether a bereavement message was appropriate, whether a joke lands in Marathi) are not automatable at any agreement you would accept. Sample these for human review and stop pretending."}]]},{"t":"p","runs":[{"text":"Never accept the fifth option before trying the first."}]},{"t":"h2","id":"re-calibrate-on-a-schedule-and-on-a-trigger","runs":[{"text":"Re-calibrate on a schedule and on a trigger"}]},{"t":"p","runs":[{"text":"A calibration goes stale. Re-run the frozen calibration set:"}]},{"t":"ul","items":[[{"text":"monthly","b":true},{"text":", as routine"}],[{"text":"whenever the judge model version changes","b":true},{"text":", including when a provider silently updates an alias like "},{"text":"-latest","code":true},{"text":", which is the argument for pinning an exact version string in the judge as much as in the system"}],[{"text":"whenever the judge prompt changes","b":true},{"text":", even trivially"}],[{"text":"whenever the input distribution moves","b":true},{"text":", such as launching in a new language or a new customer segment"}]]},{"t":"p","runs":[{"text":"This takes ten minutes because the labels already exist. Skipping it is how a team ends up six months into trusting a judge that stopped agreeing with them in March."}]},{"t":"p","runs":[{"text":"Record each run: date, judge model version, judge prompt version, kappa, per-class recall. Five lines in a file. That file is the provenance of every quality number your organisation believes."}]},{"t":"h2","id":"the-bias-you-cannot-calibrate-away","runs":[{"text":"The bias you cannot calibrate away"}]},{"t":"p","runs":[{"text":"One caveat to end on. Calibration proves the judge agrees with your labels. It does not prove your labels are right. If your two labellers share an assumption (that longer answers are more thorough, that formal register is more professional) the judge will learn that assumption and the calibration will look excellent."}]},{"t":"p","runs":[{"text":"The only defence is periodically bringing in somebody outside the team to label 20 items cold and comparing. It is uncomfortable and it is the cheapest audit available."}]}],"check":{"question":[{"text":"A judge is validated on 200 randomly sampled production items and agrees with human labels 94% of the time. Policy violations occur in about 3% of traffic. Why is this validation weak?"}],"options":[{"text":[{"text":"A random sample of 200 contains roughly six violations, so the figure says almost nothing about whether the judge detects the thing it exists to detect."}],"correct":true,"tempting":null},{"text":[{"text":"200 items is too small for any reliable agreement estimate; at least 1,000 are needed."}],"correct":false,"tempting":[{"text":"Blames overall sample size when the shortage is specifically of positive cases, which stratifying fixes without any increase in labelling effort."}]},{"text":[{"text":"Human labels on production data are unreliable because reviewers see the model’s output before deciding."}],"correct":false,"tempting":[{"text":"Raises an anchoring risk worth controlling for, but it does not explain why a 94% figure on this sample is uninformative."}]},{"text":[{"text":"Agreement should be measured against a second judge model rather than humans, to remove human inconsistency."}],"correct":false,"tempting":[{"text":"Replaces the ground truth with another uncalibrated model, which measures whether two models share a bias rather than whether either is right."}]}],"order":[2,3,0,1],"why":null},"prev":{"slug":"reference-answers-and-equivalence","title":"Gold answers, and asking whether two answers agree","minutes":9},"next":{"slug":"ranking-systems-by-preference","title":"Turning pairwise wins into a ranking you can defend","minutes":9}}}