Pairwise human comparison, done properly

Why pairwise

Asking somebody to rate an answer out of ten produces noise: different people use different parts of the scale, the same person drifts over a session, and nobody can hold a stable standard across two hundred items.

Asking which of two answers is better produces a judgement people can actually make. It is the right instrument for everything you cannot measure automatically: tone, helpfulness, whether the length suits the question, whether a decline was appropriate.

The protocol

Six requirements, each preventing a specific error:

1. Blind. The judge must not know which system produced which answer. Obvious, and violated constantly by teams evaluating their own work.
2. Randomise the order per item, and record which side was which. Position bias is real and substantial in both human and model judges.
3. Allow a tie. Forcing a choice on equivalent answers manufactures signal from nothing. A high tie rate is itself a finding: it means your two systems are not meaningfully different.
4. State the criterion. “Which is better” gets you the judge’s private theory of better. “Which would you rather receive as a customer, considering correctness first and tone second” gets you an answer to your question.
5. Same inputs, same conditions. Both systems get identical prompts, identical retrieved context, identical sampling settings.
6. Overlap for agreement. Have two or three judges do the same 50 items. Their agreement rate is the ceiling on what this measurement can tell you: if two careful people agree only 60% of the time, no sample size rescues the comparison, and the task needs specifying better before it needs measuring.

How many comparisons

The count depends entirely on the size of the difference you would act on:

• A 65% win rate (a large, obvious difference): about 150 comparisons gives an interval that excludes 50%.
• A 60% win rate: about 380.
• A 55% win rate (a small difference): about 1,500.

That progression is worth internalising. Detecting a small difference is expensive, and the correct response is usually not to buy more comparisons but to decide that a difference that small does not change anything.

Decide the number before you start. Stopping when the result looks good is the most reliable way to produce a result that is not true.

Reporting it

n = 400 comparisons, 3 judges, pairwise agreement 0.78
tuned wins 182 · ties 96 · baseline wins 122
win rate excluding ties: 182/304 = 0.599  [95% CI 0.543 – 0.653]
mean length: tuned 148 tokens, baseline 96 tokens

That last line is the one to always include. Longer answers win more often than their quality warrants, in human judging as much as in model judging. If your tuned model’s answers are 50% longer, a meaningful part of the 60% win rate is length rather than quality.

The check: compute the win rate separately on items where the two answers were of similar length. If it collapses towards 50%, you measured verbosity.

Making it cheap

The tooling does not need to be sophisticated. A static HTML page that shows a prompt and two answers, records a keypress and appends a row to a CSV is fifty lines and works. A spreadsheet with randomised columns works. Argilla is free and open source if you want annotation management, agreement tracking and a proper interface.

The expensive part is attention, not software. Thirty to ninety seconds per comparison means 400 comparisons is six to ten hours of somebody’s careful work, and their care degrades after about an hour. Spread it, and re-check agreement on a few items from the start of the session against the end.

What it cannot tell you

A win rate says which answer people prefer when reading two side by side. That is not the same as which answer serves them better in use. People prefer confident answers to hedged ones, complete-looking ones to short ones, and agreement to correction, and in every one of those cases the preferred answer is sometimes the worse one.

So pair it with something objective wherever you can: a correctness check, a verifier, a task-completion rate from real use. Preference is one instrument, it measures one thing, and a project that measures only preference will optimise for being liked.