Proving the judge agrees with you, and re-proving it

The calibration set

A judge is a measuring instrument, and an uncalibrated instrument produces numbers, not measurements. Calibration here means one specific thing: showing that the judge’s verdicts match careful human verdicts on the same items, and stating by how much they differ.

Build a calibration set and keep it separate from everything else:

• 50 to 100 items, drawn from real traffic
• labelled independently by two people, disagreements resolved and the ruling written down
• deliberately enriched with the rare class. If violations are 4% of traffic, a random 50 contains two of them and tells you nothing about whether the judge catches violations. Take 25 known violations and 25 clean items, and remember when you report that you have stratified.
• frozen, versioned, and never used for tuning the system itself

That set costs about five hours of two people’s time once, and it is what every judge number you ever quote will rest on.

Agreement, per class

Run the judge over the calibration set and compare against the human labels. Compute overall agreement; and then immediately stop looking at it and compute the two numbers that matter.

An illustrative result on a set of 50 with 12 real violations:

• Overall agreement: 44/50, which is 88%. Sounds fine.
• Of the 12 real violations, the judge flagged 4. Recall 33%.
• Of the 6 items it flagged, 4 were real. Precision 67%.

The judge is 88% agreeable and catches one violation in three. If you had shipped a dashboard on the overall figure, you would have reported a low violation rate and been technically correct and completely wrong.

This is the same lesson as accuracy under imbalance from module two, and it bites harder here because the number feels like a validation rather than a metric.

Use Cohen’s kappa as the headline instead, for the reason given in module one: it discounts the agreement you would get by chance. Above 0.8, the judge is a usable instrument. Between 0.6 and 0.8, usable with the slack stated on every number it produces. Below 0.6, it is not measuring your criterion and no amount of prompt fiddling should be reported as if it were.

What to do when agreement is poor

In order of how often it works:

1. Split the rule. Most poor agreement comes from a compound criterion. Two rules at 0.85 beat one at 0.55.
2. Add the boundary rulings from your labelling guide directly into the judge prompt as few-shot examples. Two or three well-chosen edge cases move agreement more than any amount of instruction rewriting.
3. Require quoted evidence before the verdict, if you are not already.
4. Change judge model. A larger model helps on subtle criteria. It also costs more per call, so measure whether it earns it.
5. Accept that it is a human criterion. Some judgements (whether a bereavement message was appropriate, whether a joke lands in Marathi) are not automatable at any agreement you would accept. Sample these for human review and stop pretending.

Never accept the fifth option before trying the first.

Re-calibrate on a schedule and on a trigger

A calibration goes stale. Re-run the frozen calibration set:

• monthly, as routine
• whenever the judge model version changes, including when a provider silently updates an alias like -latest, which is the argument for pinning an exact version string in the judge as much as in the system
• whenever the judge prompt changes, even trivially
• whenever the input distribution moves, such as launching in a new language or a new customer segment

This takes ten minutes because the labels already exist. Skipping it is how a team ends up six months into trusting a judge that stopped agreeing with them in March.

Record each run: date, judge model version, judge prompt version, kappa, per-class recall. Five lines in a file. That file is the provenance of every quality number your organisation believes.

The bias you cannot calibrate away

One caveat to end on. Calibration proves the judge agrees with your labels. It does not prove your labels are right. If your two labellers share an assumption (that longer answers are more thorough, that formal register is more professional) the judge will learn that assumption and the calibration will look excellent.

The only defence is periodically bringing in somebody outside the team to label 20 items cold and comparing. It is uncomfortable and it is the cheapest audit available.