Why you will end up using one
You cannot pay humans to read 4,000 outputs a week. A model can, for a few dollars, and for many criteria it agrees with careful human labels most of the time. Judging is a real technique and you should use it.
It also fails in ways that are specific, repeatable, and easy to miss — because the failures produce plausible numbers, not errors.
The four biases worth knowing by name
Position bias. In pairwise comparisons ("which reply is better, A or B?"), judges systematically favour one position. The fix is cheap: run every pair twice, with the order swapped, and count a win only when both orderings agree. The rate at which the two orderings *disagree* is your noise floor — if the judge contradicts itself on 25% of pairs, no win margin under 25 points means anything.
Length bias. Longer, more structured, more headed answers score higher, largely regardless of content. This matters because the easiest way to change a prompt is to make it produce more words. Check it directly: plot judge score against output length across your set. If the correlation is strong, your "quality improvement" may be a verbosity improvement, and your users may hate it.
Self-preference. Models tend to score outputs from their own family more highly. Where you can, judge with a model from a different provider than the one generating. Never let the same model be both the generator and the sole arbiter of its own comparison.
Scale compression. Ask for a 1-to-10 rating and you will get sevens and eights, forever, on everything. The information content is near zero. Ask a specific yes/no question with a reason and you get data you can use.
You are checking one support reply against one rule.
RULE: The reply must not state or imply a refund deadline
that is not present in the policy text below.
POLICY: {policy}
REPLY: {reply}
First write one sentence of reasoning quoting the relevant
span of the reply. Then output strict JSON:
{"reason": "...", "violates_rule": true | false}Reasoning before the verdict, one rule per call, a boolean out. Asking for five judgements in one call makes all five worse and correlates their errors.
The step almost everyone skips
Calibrate the judge against your own labels.
Take 50 items you and a colleague labelled by hand in lesson two. Run the judge on them. Compute agreement.
- Two humans agreed 95% of the time. The judge agrees with the humans 92%. Use it, and remember every number it gives you carries roughly eight points of error, so a five-point improvement is not a result.
- The judge agrees 71%. It is not usable for this criterion. Rewrite the rubric, try one rule per call, try a different model — and re-measure. Do not ship a dashboard built on it.
A judge you have not calibrated is not a measurement instrument. It is a confident opinion with a decimal point on it.
Two things judges are good at, one they are not
They are good at rule checks — did this reply promise something it should not, did it stay in the right language, did it cite a document that exists. They are good at pairwise preference on a fixed rubric, once you have controlled for position.
They are poor at absolute scores you plan to compare across time or across teams. Judge scores drift when the judge model updates, and "our agent scores 8.4" is meaningless to anyone outside your repository. Keep judge numbers as internal, relative, same-run comparisons. That is what they are honest for.
Before you move on