A model judge you have checked against people

The layer above assertions

An answer passes every mechanical check and is still wrong. Someone has to read it. At fifty cases that someone can be you; at five hundred, or at every commit, it has to be a model. Using a model to grade a model is standard practice now, and it works: under conditions that most teams skip. The evaluation course devotes a block to the biases of judges; this lesson is what you must do before trusting one on a feature you are shipping.

Write the rubric as questions with yes-or-no answers

A judge asked for a score out of ten returns numbers that drift with phrasing, cluster around seven, and mean nothing across prompt versions. A judge asked binary questions returns verdicts you can count.

For the answer below, reply with JSON: {"correct_window": bool,
"applies_from_purchase": bool, "no_invented_policy": bool,
"reason": "one sentence"}
correct_window: does the answer state the refund window is 14 days?
applies_from_purchase: does it say the window runs from purchase, not renewal?
no_invented_policy: does it avoid stating any policy not in the reference?

Each criterion is a fact about the answer. The case passes if all required criteria are true. The reason field is not decoration: it is what you read when the verdict looks wrong, and it is how you find rubric criteria the judge is misreading.

Pairwise when there is no reference

For open-ended output (a draft email, a summary) there is often no reference answer, and “is this good?” is unanswerable. “Is A better than B?” is answerable. Show the judge two outputs, from the old prompt and the new, and ask which better satisfies the rubric. A comparison is easier and more consistent than an absolute grade, and it is what you actually want to know when deciding whether to ship a change.

Position bias, and the swap

Judges prefer the first option shown, or the second, by a margin that varies by model and is never zero. The test is simple: run every pair both ways. If the judge prefers A in the A-then-B order and B in the B-then-A order, the verdict is position, not quality, and that pair is a tie. Report the tie rate; if it is high, the two prompts are not different enough to distinguish, which is a finding.

Self-preference

A model judging its own outputs favours them: the same style, the same phrasing choices read as correct to it. Judge with a different model family from the one generating, or with a stronger model from the same family if that is all you have, and say which in the report.

The calibration step, which is the whole point

Before the judge grades anything that decides a release, grade fifty answers by hand (two people, independently, then reconcile) and run the judge on the same fifty. Compare. Simple agreement, the share of answers where judge and humans reached the same verdict, is the number to write down; the evaluation course covers the chance-corrected version. Above ninety per cent, use the judge and quote the figure. Between eighty and ninety, use it for triage but not for decisions. Below eighty, the rubric is unclear or the task is beyond the judge, and every number it produces afterwards is noise with a decimal point.

Redo this whenever the rubric, the judge model or the feature changes materially. Agreement measured in March against a judge model that was retired in June is not a figure you can quote in July.

Cost per run

A judge call per case, with the rubric, the reference and the answer, is a few thousand tokens. Fifty cases is a few hundred thousand tokens: well under a dollar with a mid-sized model, a few dollars with a frontier one. Pairwise doubles it; the order swap doubles it again. Cheap enough to run on every prompt change; expensive enough that the assertion layer should filter first, so the judge only reads answers that were not mechanically broken.

What a judge cannot know

A judge sees the answer and whatever reference you gave it. It does not know whether the reference is right, whether the corpus has changed, or whether the user’s real question was different from the one typed. It grades against the rubric, and a rubric that encodes a mistaken policy will pass answers that repeat the mistake. That is why the error-analysis lesson later in this block keeps “expectation wrong” as a category, and why a judge’s verdict is evidence to be examined rather than a decision made.

Reading the disagreements

The calibration sample’s disagreements are the most useful artefact this lesson produces. Each one is either a rubric criterion the judge misreads, a case whose expectation is wrong, or a genuine hard call. Sorting the fifty into those three piles takes an hour and improves the rubric more than any amount of prompt engineering on the judge itself.