{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"judging-with-a-model","title":"Using a model to mark the homework","url":"https://addaly.com/learn/context-engineering/judging-with-a-model","position":83,"of":87,"minutes":9,"answer":"A model judge is the only practical way to score open-ended answers at volume, and it carries position.","keyIdea":"A model judge is the only practical way to score open-ended answers at volume, and it carries position, verbosity and self-preference biases that must be controlled for and an agreement rate with humans that must be measured before any of its numbers mean anything."},"body":{"format":"markdown","text":"## When you actually need one\n\nMost of what a golden set checks needs no judge. A cited identifier, a required substring, a refusal, a number, a schema field: all mechanical, all free, all exact. Push everything you can into that category first, because a check that costs nothing runs on every commit and a check that costs money runs monthly.\n\nWhat is left is open-ended: is this summary faithful to the source, is this explanation clear, does this answer address what was asked. Human labelling is the gold standard and does not scale past a few hundred items a week. A model judge is the practical instrument, and it is a biased one.\n\n## The three biases, and the fixes\n\n**Position bias.** Shown two answers and asked which is better, judges systematically favour one position: usually the first. The effect is large enough to reverse verdicts on close pairs.\n\nThe fix is complete and cheap: run every comparison twice, with the order swapped, and count a win only when both orders agree. When they disagree, score it a tie. This doubles the cost and it is not optional; a pairwise judge run in one order is measuring position as much as quality.\n\n**Verbosity bias.** Longer answers score higher, independent of content. This one is dangerous because it interacts with what you are testing: a prompt change that makes answers longer will look like an improvement. Control for it by checking whether the length distributions differ between variants, and be suspicious of any win that arrives alongside a length increase. Instructing the judge to disregard length helps somewhat and does not remove it.\n\n**Self-preference.** Models tend to rate their own outputs above others'. Where it matters, judge with a different model family from the one being evaluated.\n\n## Calibrate, or the number is decoration\n\nBefore trusting a judge, find out whether it agrees with you.\n\nLabel 100 items by hand. Run the judge on the same 100. Compute agreement. Raw percentage is fine to start with, though Cohen's kappa is better because it accounts for agreement by chance: with two categories and an unbalanced set, 80 percent raw agreement can be close to worthless.\n\nThen read the number for what it implies about resolution. A judge agreeing with humans 70 percent of the time is making a wrong call on three items in ten, and it cannot reliably detect a three-point difference between two variants, because its own error is larger than the effect. Knowing that saves you from a quarter of decisions made on noise.\n\nIf agreement is poor, the usual cause is the rubric rather than the model, and the usual cure is specificity: replace \"rate helpfulness from 1 to 10\" with a small number of concrete, checkable questions.\n\n## Pairwise beats scoring\n\nAsking for a 1-to-10 score produces a distribution clustered at 7 and 8, with almost no discrimination in the region you care about, and the scale drifts between runs so yesterday's 7 is not today's 7.\n\nAsking \"which of these two is better, and why\" is a much easier judgement and a much more stable one. It also fits the actual decision: you are comparing two variants, not certifying an absolute quality level.\n\nWhere you do need absolute scores, make them binary and specific. \"Is every factual claim supported by the provided passages: yes or no\" is answerable. \"Rate faithfulness out of 10\" is not.\n\n## Make the judge explain first\n\nPut the reasoning field before the verdict field in the schema, for the reason the constrained-decoding lesson gave: a verdict generated first is a guess the explanation then rationalises.\n\nThe explanations have a second use that is worth more than the scores. Reading thirty judge explanations for the cases your system failed is the fastest qualitative diagnosis available, and it regularly surfaces failure modes nobody had named, which then become their own mechanical check, and stop needing a judge at all.\n\n## The free path\n\nJudging can be expensive: two orders, several hundred items, every run. Three things keep it affordable.\n\nRoute judging through a **batch endpoint**, since nobody is waiting. Roughly half price for work that runs overnight.\n\nUse a **small model** where the rubric is simple. Faithfulness against a provided passage is close to an entailment task, and small open models handle it respectably. Calibrate it against your human labels the same way; if a 7-billion-parameter model running locally agrees with you as often as a frontier model does, it is the correct choice and it costs nothing per item.\n\n**Judge a sample.** You do not need a judge on all 300 items on every commit. Mechanical checks on everything, judged scores on a fixed 100, full runs before a release.\n\n## The limit to state out loud\n\nA judge measures agreement with a rubric, not truth. If the rubric is wrong, the judge is confidently and consistently wrong, and consistency reads as reliability on a dashboard.\n\nSo keep a small human-labelled sample running alongside, permanently, and watch the agreement rate as its own metric. When it drifts, the judge has stopped measuring what you think it measures. And that drift will not announce itself in the scores, which will carry on looking perfectly steady."},"check":{"question":"A pairwise judge is run once per comparison, always presenting the new variant second. What does the resulting win rate partly measure?","options":[{"text":"The judge's positional preference, which favours one slot regardless of content.","correct":true},{"text":"The judge's self-preference, since one of the two answers came from a related model family.","correct":false,"tempting":"A real bias, but it depends on which model generated each answer rather than on the order they were shown in."},{"text":"The verbosity of the second answer, because later positions receive more of the judge's attention budget.","correct":false,"tempting":"Verbosity bias is about length rather than slot, and it operates wherever the longer answer sits."},{"text":"The judge's calibration against human labels, which is unknown until a sample is hand-scored.","correct":false,"tempting":"Calibration bounds how much any verdict is worth; it is not the specific artefact introduced by fixing the order."}]},"prev":{"slug":"is-the-difference-real","title":"Three points better, or three points of noise","url":"https://addaly.com/learn/context-engineering/is-the-difference-real"},"next":{"slug":"logging-a-request-you-can-replay","title":"Logging enough to rebuild the request","url":"https://addaly.com/learn/context-engineering/logging-a-request-you-can-replay"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Using a model to mark the homework', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/judging-with-a-model (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Using a model to mark the homework","courseTitle":"Context Engineering","module":{"index":9,"slug":"shipping-and-measuring","title":"Shipping context, and proving it got better"},"answer":null,"keyIdea":"A model judge is the only practical way to score open-ended answers at volume, and it carries position, verbosity and self-preference biases that must be controlled for and an agreement rate with humans that must be measured before any of its numbers mean anything.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"when-you-actually-need-one","runs":[{"text":"When you actually need one"}]},{"t":"p","runs":[{"text":"Most of what a golden set checks needs no judge. A cited identifier, a required substring, a refusal, a number, a schema field: all mechanical, all free, all exact. Push everything you can into that category first, because a check that costs nothing runs on every commit and a check that costs money runs monthly."}]},{"t":"p","runs":[{"text":"What is left is open-ended: is this summary faithful to the source, is this explanation clear, does this answer address what was asked. Human labelling is the gold standard and does not scale past a few hundred items a week. A model judge is the practical instrument, and it is a biased one."}]},{"t":"h2","id":"the-three-biases-and-the-fixes","runs":[{"text":"The three biases, and the fixes"}]},{"t":"p","runs":[{"text":"Position bias.","b":true},{"text":" Shown two answers and asked which is better, judges systematically favour one position: usually the first. The effect is large enough to reverse verdicts on close pairs."}]},{"t":"p","runs":[{"text":"The fix is complete and cheap: run every comparison twice, with the order swapped, and count a win only when both orders agree. When they disagree, score it a tie. This doubles the cost and it is not optional; a pairwise judge run in one order is measuring position as much as quality."}]},{"t":"p","runs":[{"text":"Verbosity bias.","b":true},{"text":" Longer answers score higher, independent of content. This one is dangerous because it interacts with what you are testing: a prompt change that makes answers longer will look like an improvement. Control for it by checking whether the length distributions differ between variants, and be suspicious of any win that arrives alongside a length increase. Instructing the judge to disregard length helps somewhat and does not remove it."}]},{"t":"p","runs":[{"text":"Self-preference.","b":true},{"text":" Models tend to rate their own outputs above others’. Where it matters, judge with a different model family from the one being evaluated."}]},{"t":"h2","id":"calibrate-or-the-number-is-decoration","runs":[{"text":"Calibrate, or the number is decoration"}]},{"t":"p","runs":[{"text":"Before trusting a judge, find out whether it agrees with you."}]},{"t":"p","runs":[{"text":"Label 100 items by hand. Run the judge on the same 100. Compute agreement. Raw percentage is fine to start with, though Cohen’s kappa is better because it accounts for agreement by chance: with two categories and an unbalanced set, 80 percent raw agreement can be close to worthless."}]},{"t":"p","runs":[{"text":"Then read the number for what it implies about resolution. A judge agreeing with humans 70 percent of the time is making a wrong call on three items in ten, and it cannot reliably detect a three-point difference between two variants, because its own error is larger than the effect. Knowing that saves you from a quarter of decisions made on noise."}]},{"t":"p","runs":[{"text":"If agreement is poor, the usual cause is the rubric rather than the model, and the usual cure is specificity: replace “rate helpfulness from 1 to 10” with a small number of concrete, checkable questions."}]},{"t":"h2","id":"pairwise-beats-scoring","runs":[{"text":"Pairwise beats scoring"}]},{"t":"p","runs":[{"text":"Asking for a 1-to-10 score produces a distribution clustered at 7 and 8, with almost no discrimination in the region you care about, and the scale drifts between runs so yesterday’s 7 is not today’s 7."}]},{"t":"p","runs":[{"text":"Asking “which of these two is better, and why” is a much easier judgement and a much more stable one. It also fits the actual decision: you are comparing two variants, not certifying an absolute quality level."}]},{"t":"p","runs":[{"text":"Where you do need absolute scores, make them binary and specific. “Is every factual claim supported by the provided passages: yes or no” is answerable. “Rate faithfulness out of 10” is not."}]},{"t":"h2","id":"make-the-judge-explain-first","runs":[{"text":"Make the judge explain first"}]},{"t":"p","runs":[{"text":"Put the reasoning field before the verdict field in the schema, for the reason the constrained-decoding lesson gave: a verdict generated first is a guess the explanation then rationalises."}]},{"t":"p","runs":[{"text":"The explanations have a second use that is worth more than the scores. Reading thirty judge explanations for the cases your system failed is the fastest qualitative diagnosis available, and it regularly surfaces failure modes nobody had named, which then become their own mechanical check, and stop needing a judge at all."}]},{"t":"h2","id":"the-free-path","runs":[{"text":"The free path"}],"role":"free"},{"t":"p","runs":[{"text":"Judging can be expensive: two orders, several hundred items, every run. Three things keep it affordable."}]},{"t":"p","runs":[{"text":"Route judging through a "},{"text":"batch endpoint","b":true},{"text":", since nobody is waiting. Roughly half price for work that runs overnight."}]},{"t":"p","runs":[{"text":"Use a "},{"text":"small model","b":true},{"text":" where the rubric is simple. Faithfulness against a provided passage is close to an entailment task, and small open models handle it respectably. Calibrate it against your human labels the same way; if a 7-billion-parameter model running locally agrees with you as often as a frontier model does, it is the correct choice and it costs nothing per item."}]},{"t":"p","runs":[{"text":"Judge a sample.","b":true},{"text":" You do not need a judge on all 300 items on every commit. Mechanical checks on everything, judged scores on a fixed 100, full runs before a release."}]},{"t":"h2","id":"the-limit-to-state-out-loud","runs":[{"text":"The limit to state out loud"}],"role":"limit"},{"t":"p","runs":[{"text":"A judge measures agreement with a rubric, not truth. If the rubric is wrong, the judge is confidently and consistently wrong, and consistency reads as reliability on a dashboard."}]},{"t":"p","runs":[{"text":"So keep a small human-labelled sample running alongside, permanently, and watch the agreement rate as its own metric. When it drifts, the judge has stopped measuring what you think it measures. And that drift will not announce itself in the scores, which will carry on looking perfectly steady."}]}],"check":{"question":[{"text":"A pairwise judge is run once per comparison, always presenting the new variant second. What does the resulting win rate partly measure?"}],"options":[{"text":[{"text":"The judge’s positional preference, which favours one slot regardless of content."}],"correct":true,"tempting":null},{"text":[{"text":"The judge’s self-preference, since one of the two answers came from a related model family."}],"correct":false,"tempting":[{"text":"A real bias, but it depends on which model generated each answer rather than on the order they were shown in."}]},{"text":[{"text":"The verbosity of the second answer, because later positions receive more of the judge’s attention budget."}],"correct":false,"tempting":[{"text":"Verbosity bias is about length rather than slot, and it operates wherever the longer answer sits."}]},{"text":[{"text":"The judge’s calibration against human labels, which is unknown until a sample is hand-scored."}],"correct":false,"tempting":[{"text":"Calibration bounds how much any verdict is worth; it is not the specific artefact introduced by fixing the order."}]}],"order":[3,1,2,0],"why":null},"prev":{"slug":"is-the-difference-real","title":"Three points better, or three points of noise","minutes":10},"next":{"slug":"logging-a-request-you-can-replay","title":"Logging enough to rebuild the request","minutes":9}}}