{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"the-judge-needs-judging","title":"LLM-as-judge, and where it misleads you","url":"https://addaly.com/learn/evaluating-ai/the-judge-needs-judging","position":40,"of":76,"minutes":9,"answer":"A judge you have not measured against your own labels is an opinion with a decimal point.","keyIdea":"A judge you have not measured against your own labels is an opinion with a decimal point."},"body":{"format":"markdown","text":"## Why you will end up using one\n\nYou cannot pay humans to read 4,000 outputs a week. A model can, for a few dollars, and for many criteria it agrees with careful human labels most of the time. Judging is a real technique and you should use it.\n\nIt also fails in ways that are specific, repeatable, and easy to miss, because the failures produce plausible numbers, not errors.\n\n## The four biases worth knowing by name\n\n**Position bias.** In pairwise comparisons (\"which reply is better, A or B?\"), judges systematically favour one position. The fix is cheap: run every pair twice, with the order swapped, and count a win only when both orderings agree. The rate at which the two orderings *disagree* is your noise floor: if the judge contradicts itself on 25% of pairs, no win margin under 25 points means anything.\n\n**Length bias.** Longer, more structured, more headed answers score higher, largely regardless of content. This matters because the easiest way to change a prompt is to make it produce more words. Check it directly: plot judge score against output length across your set. If the correlation is strong, your \"quality improvement\" may be a verbosity improvement, and your users may hate it.\n\n**Self-preference.** Models tend to score outputs from their own family more highly. Where you can, judge with a model from a different provider than the one generating. Never let the same model be both the generator and the sole arbiter of its own comparison.\n\n**Scale compression.** Ask for a 1-to-10 rating and you will get sevens and eights, forever, on everything. The information content is near zero. Ask a specific yes/no question with a reason and you get data you can use.\n\n```\nYou are checking one support reply against one rule.\n\nRULE: The reply must not state or imply a refund deadline\n      that is not present in the policy text below.\n\nPOLICY: {policy}\nREPLY:  {reply}\n\nFirst write one sentence of reasoning quoting the relevant\nspan of the reply. Then output strict JSON:\n{\"reason\": \"...\", \"violates_rule\": true | false}\n```\n\nReasoning before the verdict, one rule per call, a boolean out. Asking for five judgements in one call makes all five worse and correlates their errors.\n\n## The step almost everyone skips\n\nCalibrate the judge against your own labels.\n\nTake 50 items you and a colleague labelled by hand in lesson two. Run the judge on them. Compute agreement.\n\n- Two humans agreed 95% of the time. The judge agrees with the humans 92%. Use it, and remember every number it gives you carries roughly eight points of error, so a five-point improvement is not a result.\n- The judge agrees 71%. It is not usable for this criterion. Rewrite the rubric, try one rule per call, try a different model, and re-measure. Do not ship a dashboard built on it.\n\nA judge you have not calibrated is not a measurement instrument. It is a confident opinion with a decimal point on it.\n\n## Two things judges are good at, one they are not\n\nThey are good at **rule checks**: did this reply promise something it should not, did it stay in the right language, did it cite a document that exists. They are good at **pairwise preference on a fixed rubric**, once you have controlled for position.\n\nThey are poor at **absolute scores you plan to compare across time or across teams**. Judge scores drift when the judge model updates, and \"our agent scores 8.4\" is meaningless to anyone outside your repository. Keep judge numbers as internal, relative, same-run comparisons. That is what they are honest for."},"check":{"question":"A pairwise judge is shown the old and new prompt's answers and picks the new one 68% of the time across 200 real inputs. The team is ready to ship. What is the strongest reason to pause?","options":[{"text":"Each pair was shown in one order only, so an unknown part of that 68% could be the judge preferring whichever answer it read first.","correct":true},{"text":"68% is too close to a coin flip to support a decision at any sample size.","correct":false,"tempting":"Focuses on effect size alone and misses that a genuine 68% preference over 200 inputs would be a substantial, shippable win."},{"text":"Pairwise judging is inherently unstable; they should score each answer 1 to 10 and compare the means.","correct":false,"tempting":"Swaps to an absolute scale that judges use far less consistently than a forced choice between two options."},{"text":"A human should read the 32% of cases the new prompt lost before anything ships.","correct":false,"tempting":"Proposes a reasonable-sounding review of the losses while leaving the measurement's own bias completely unexamined."}]},"prev":{"slug":"evaluating-tabular-models","title":"Tables, cross-validation, and the leak that gives you 0.99","url":"https://addaly.com/learn/evaluating-ai/evaluating-tabular-models"},"next":{"slug":"writing-a-rubric-a-machine-can-follow","title":"Turning a quality you can feel into rules a machine can apply","url":"https://addaly.com/learn/evaluating-ai/writing-a-rubric-a-machine-can-follow"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'LLM-as-judge, and where it misleads you', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/the-judge-needs-judging (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"LLM-as-judge, and where it misleads you","courseTitle":"Knowing If It Works","module":{"index":5,"slug":"automating-judgement","title":"Automating judgement"},"answer":null,"keyIdea":"A judge you have not measured against your own labels is an opinion with a decimal point.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-you-will-end-up-using-one","runs":[{"text":"Why you will end up using one"}]},{"t":"p","runs":[{"text":"You cannot pay humans to read 4,000 outputs a week. A model can, for a few dollars, and for many criteria it agrees with careful human labels most of the time. Judging is a real technique and you should use it."}]},{"t":"p","runs":[{"text":"It also fails in ways that are specific, repeatable, and easy to miss, because the failures produce plausible numbers, not errors."}]},{"t":"h2","id":"the-four-biases-worth-knowing-by-name","runs":[{"text":"The four biases worth knowing by name"}]},{"t":"p","runs":[{"text":"Position bias.","b":true},{"text":" In pairwise comparisons (“which reply is better, A or B?”), judges systematically favour one position. The fix is cheap: run every pair twice, with the order swapped, and count a win only when both orderings agree. The rate at which the two orderings "},{"text":"disagree","i":true},{"text":" is your noise floor: if the judge contradicts itself on 25% of pairs, no win margin under 25 points means anything."}]},{"t":"p","runs":[{"text":"Length bias.","b":true},{"text":" Longer, more structured, more headed answers score higher, largely regardless of content. This matters because the easiest way to change a prompt is to make it produce more words. Check it directly: plot judge score against output length across your set. If the correlation is strong, your “quality improvement” may be a verbosity improvement, and your users may hate it."}]},{"t":"p","runs":[{"text":"Self-preference.","b":true},{"text":" Models tend to score outputs from their own family more highly. Where you can, judge with a model from a different provider than the one generating. Never let the same model be both the generator and the sole arbiter of its own comparison."}]},{"t":"p","runs":[{"text":"Scale compression.","b":true},{"text":" Ask for a 1-to-10 rating and you will get sevens and eights, forever, on everything. The information content is near zero. Ask a specific yes/no question with a reason and you get data you can use."}]},{"t":"code","lang":"","label":"Text","text":"You are checking one support reply against one rule.\n\nRULE: The reply must not state or imply a refund deadline\n      that is not present in the policy text below.\n\nPOLICY: {policy}\nREPLY:  {reply}\n\nFirst write one sentence of reasoning quoting the relevant\nspan of the reply. Then output strict JSON:\n{\"reason\": \"...\", \"violates_rule\": true | false}"},{"t":"p","runs":[{"text":"Reasoning before the verdict, one rule per call, a boolean out. Asking for five judgements in one call makes all five worse and correlates their errors."}]},{"t":"h2","id":"the-step-almost-everyone-skips","runs":[{"text":"The step almost everyone skips"}]},{"t":"p","runs":[{"text":"Calibrate the judge against your own labels."}]},{"t":"p","runs":[{"text":"Take 50 items you and a colleague labelled by hand in lesson two. Run the judge on them. Compute agreement."}]},{"t":"ul","items":[[{"text":"Two humans agreed 95% of the time. The judge agrees with the humans 92%. Use it, and remember every number it gives you carries roughly eight points of error, so a five-point improvement is not a result."}],[{"text":"The judge agrees 71%. It is not usable for this criterion. Rewrite the rubric, try one rule per call, try a different model, and re-measure. Do not ship a dashboard built on it."}]]},{"t":"p","runs":[{"text":"A judge you have not calibrated is not a measurement instrument. It is a confident opinion with a decimal point on it."}]},{"t":"h2","id":"two-things-judges-are-good-at-one-they-are-not","runs":[{"text":"Two things judges are good at, one they are not"}]},{"t":"p","runs":[{"text":"They are good at "},{"text":"rule checks","b":true},{"text":": did this reply promise something it should not, did it stay in the right language, did it cite a document that exists. They are good at "},{"text":"pairwise preference on a fixed rubric","b":true},{"text":", once you have controlled for position."}]},{"t":"p","runs":[{"text":"They are poor at "},{"text":"absolute scores you plan to compare across time or across teams","b":true},{"text":". Judge scores drift when the judge model updates, and “our agent scores 8.4” is meaningless to anyone outside your repository. Keep judge numbers as internal, relative, same-run comparisons. That is what they are honest for."}]}],"check":{"question":[{"text":"A pairwise judge is shown the old and new prompt’s answers and picks the new one 68% of the time across 200 real inputs. The team is ready to ship. What is the strongest reason to pause?"}],"options":[{"text":[{"text":"Each pair was shown in one order only, so an unknown part of that 68% could be the judge preferring whichever answer it read first."}],"correct":true,"tempting":null},{"text":[{"text":"68% is too close to a coin flip to support a decision at any sample size."}],"correct":false,"tempting":[{"text":"Focuses on effect size alone and misses that a genuine 68% preference over 200 inputs would be a substantial, shippable win."}]},{"text":[{"text":"Pairwise judging is inherently unstable; they should score each answer 1 to 10 and compare the means."}],"correct":false,"tempting":[{"text":"Swaps to an absolute scale that judges use far less consistently than a forced choice between two options."}]},{"text":[{"text":"A human should read the 32% of cases the new prompt lost before anything ships."}],"correct":false,"tempting":[{"text":"Proposes a reasonable-sounding review of the losses while leaving the measurement’s own bias completely unexamined."}]}],"order":[2,1,0,3],"why":null},"prev":{"slug":"evaluating-tabular-models","title":"Tables, cross-validation, and the leak that gives you 0.99","minutes":9},"next":{"slug":"writing-a-rubric-a-machine-can-follow","title":"Turning a quality you can feel into rules a machine can apply","minutes":9}}}