{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"pairwise-human-comparison","title":"Pairwise human comparison, done properly","url":"https://addaly.com/learn/fine-tuning/pairwise-human-comparison","position":70,"of":84,"minutes":9,"answer":"Blind, order-randomised, tie-allowing comparisons with a stated criterion and a measured judge-agreement rate, with the number of comparisons fixed in advance from the effect size you would act on.","keyIdea":"Blind, order-randomised, tie-allowing comparisons with a stated criterion and a measured judge-agreement rate, with the number of comparisons fixed in advance from the effect size you would act on, and mean response length reported beside the win rate because length buys wins it did not earn."},"body":{"format":"markdown","text":"## Why pairwise\n\nAsking somebody to rate an answer out of ten produces noise: different people use different parts of the scale, the same person drifts over a session, and nobody can hold a stable standard across two hundred items.\n\nAsking which of two answers is better produces a judgement people can actually make. It is the right instrument for everything you cannot measure automatically: tone, helpfulness, whether the length suits the question, whether a decline was appropriate.\n\n## The protocol\n\nSix requirements, each preventing a specific error:\n\n1. **Blind.** The judge must not know which system produced which answer. Obvious, and violated constantly by teams evaluating their own work.\n2. **Randomise the order** per item, and record which side was which. Position bias is real and substantial in both human and model judges.\n3. **Allow a tie.** Forcing a choice on equivalent answers manufactures signal from nothing. A high tie rate is itself a finding: it means your two systems are not meaningfully different.\n4. **State the criterion.** \"Which is better\" gets you the judge's private theory of better. \"Which would you rather receive as a customer, considering correctness first and tone second\" gets you an answer to your question.\n5. **Same inputs, same conditions.** Both systems get identical prompts, identical retrieved context, identical sampling settings.\n6. **Overlap for agreement.** Have two or three judges do the same 50 items. Their agreement rate is the ceiling on what this measurement can tell you: if two careful people agree only 60% of the time, no sample size rescues the comparison, and the task needs specifying better before it needs measuring.\n\n## How many comparisons\n\nThe count depends entirely on the size of the difference you would act on:\n\n- **A 65% win rate** (a large, obvious difference): about 150 comparisons gives an interval that excludes 50%.\n- **A 60% win rate**: about 380.\n- **A 55% win rate** (a small difference): about 1,500.\n\nThat progression is worth internalising. Detecting a small difference is expensive, and the correct response is usually not to buy more comparisons but to decide that a difference that small does not change anything.\n\nDecide the number *before* you start. Stopping when the result looks good is the most reliable way to produce a result that is not true.\n\n## Reporting it\n\n```\nn = 400 comparisons, 3 judges, pairwise agreement 0.78\ntuned wins 182 · ties 96 · baseline wins 122\nwin rate excluding ties: 182/304 = 0.599  [95% CI 0.543 – 0.653]\nmean length: tuned 148 tokens, baseline 96 tokens\n```\n\nThat last line is the one to always include. Longer answers win more often than their quality warrants, in human judging as much as in model judging. If your tuned model's answers are 50% longer, a meaningful part of the 60% win rate is length rather than quality.\n\nThe check: compute the win rate separately on items where the two answers were of similar length. If it collapses towards 50%, you measured verbosity.\n\n## Making it cheap\n\nThe tooling does not need to be sophisticated. A static HTML page that shows a prompt and two answers, records a keypress and appends a row to a CSV is fifty lines and works. A spreadsheet with randomised columns works. Argilla is free and open source if you want annotation management, agreement tracking and a proper interface.\n\nThe expensive part is attention, not software. Thirty to ninety seconds per comparison means 400 comparisons is six to ten hours of somebody's careful work, and their care degrades after about an hour. Spread it, and re-check agreement on a few items from the start of the session against the end.\n\n## What it cannot tell you\n\nA win rate says which answer people prefer when reading two side by side. That is not the same as which answer serves them better in use. People prefer confident answers to hedged ones, complete-looking ones to short ones, and agreement to correction, and in every one of those cases the preferred answer is sometimes the worse one.\n\nSo pair it with something objective wherever you can: a correctness check, a verifier, a task-completion rate from real use. Preference is one instrument, it measures one thing, and a project that measures only preference will optimise for being liked."},"check":{"question":"A tuned model wins 60% of 400 blind pairwise comparisons. Its answers average 148 tokens against the baseline's 96. What should be checked before accepting the result?","options":[{"text":"Whether the judges were consistent, by recomputing the win rate per judge and discarding outliers.","correct":false,"tempting":"Proposes a reasonable consistency check, but discarding judges who disagree removes exactly the signal that agreement rates are meant to expose."},{"text":"Whether 400 comparisons is enough, since a 60% win rate needs about 1,500 to exclude 50%.","correct":false,"tempting":"Misapplies the sample-size guidance; 400 is ample for a 60% rate, and 1,500 is what a 55% rate would require."},{"text":"Whether the win rate holds on length-matched items, since length buys preference on its own.","correct":true},{"text":"Whether the tie rate was high, since ties reduce the effective sample and widen the interval.","correct":false,"tempting":"Names a genuine effect on precision that does not threaten the validity of the comparison the way an uncontrolled length difference does."}]},"prev":{"slug":"a-held-out-set-that-holds","title":"A held-out set that actually holds","url":"https://addaly.com/learn/fine-tuning/a-held-out-set-that-holds"},"next":{"slug":"judging-a-model-you-distilled","title":"Judging a model you distilled from the judge","url":"https://addaly.com/learn/fine-tuning/judging-a-model-you-distilled"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Pairwise human comparison, done properly', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/pairwise-human-comparison (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Pairwise human comparison, done properly","courseTitle":"Fine-Tuning, and When Not To","module":{"index":8,"slug":"did-it-actually-work","title":"Did it actually work"},"answer":null,"keyIdea":"Blind, order-randomised, tie-allowing comparisons with a stated criterion and a measured judge-agreement rate, with the number of comparisons fixed in advance from the effect size you would act on, and mean response length reported beside the win rate because length buys wins it did not earn.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-pairwise","runs":[{"text":"Why pairwise"}]},{"t":"p","runs":[{"text":"Asking somebody to rate an answer out of ten produces noise: different people use different parts of the scale, the same person drifts over a session, and nobody can hold a stable standard across two hundred items."}]},{"t":"p","runs":[{"text":"Asking which of two answers is better produces a judgement people can actually make. It is the right instrument for everything you cannot measure automatically: tone, helpfulness, whether the length suits the question, whether a decline was appropriate."}]},{"t":"h2","id":"the-protocol","runs":[{"text":"The protocol"}]},{"t":"p","runs":[{"text":"Six requirements, each preventing a specific error:"}]},{"t":"ol","start":1,"items":[[{"text":"Blind.","b":true},{"text":" The judge must not know which system produced which answer. Obvious, and violated constantly by teams evaluating their own work."}],[{"text":"Randomise the order","b":true},{"text":" per item, and record which side was which. Position bias is real and substantial in both human and model judges."}],[{"text":"Allow a tie.","b":true},{"text":" Forcing a choice on equivalent answers manufactures signal from nothing. A high tie rate is itself a finding: it means your two systems are not meaningfully different."}],[{"text":"State the criterion.","b":true},{"text":" “Which is better” gets you the judge’s private theory of better. “Which would you rather receive as a customer, considering correctness first and tone second” gets you an answer to your question."}],[{"text":"Same inputs, same conditions.","b":true},{"text":" Both systems get identical prompts, identical retrieved context, identical sampling settings."}],[{"text":"Overlap for agreement.","b":true},{"text":" Have two or three judges do the same 50 items. Their agreement rate is the ceiling on what this measurement can tell you: if two careful people agree only 60% of the time, no sample size rescues the comparison, and the task needs specifying better before it needs measuring."}]]},{"t":"h2","id":"how-many-comparisons","runs":[{"text":"How many comparisons"}]},{"t":"p","runs":[{"text":"The count depends entirely on the size of the difference you would act on:"}]},{"t":"ul","items":[[{"text":"A 65% win rate","b":true},{"text":" (a large, obvious difference): about 150 comparisons gives an interval that excludes 50%."}],[{"text":"A 60% win rate","b":true},{"text":": about 380."}],[{"text":"A 55% win rate","b":true},{"text":" (a small difference): about 1,500."}]]},{"t":"p","runs":[{"text":"That progression is worth internalising. Detecting a small difference is expensive, and the correct response is usually not to buy more comparisons but to decide that a difference that small does not change anything."}]},{"t":"p","runs":[{"text":"Decide the number "},{"text":"before","i":true},{"text":" you start. Stopping when the result looks good is the most reliable way to produce a result that is not true."}]},{"t":"h2","id":"reporting-it","runs":[{"text":"Reporting it"}]},{"t":"code","lang":"","label":"Text","text":"n = 400 comparisons, 3 judges, pairwise agreement 0.78\ntuned wins 182 · ties 96 · baseline wins 122\nwin rate excluding ties: 182/304 = 0.599  [95% CI 0.543 – 0.653]\nmean length: tuned 148 tokens, baseline 96 tokens"},{"t":"p","runs":[{"text":"That last line is the one to always include. Longer answers win more often than their quality warrants, in human judging as much as in model judging. If your tuned model’s answers are 50% longer, a meaningful part of the 60% win rate is length rather than quality."}]},{"t":"p","runs":[{"text":"The check: compute the win rate separately on items where the two answers were of similar length. If it collapses towards 50%, you measured verbosity."}]},{"t":"h2","id":"making-it-cheap","runs":[{"text":"Making it cheap"}]},{"t":"p","runs":[{"text":"The tooling does not need to be sophisticated. A static HTML page that shows a prompt and two answers, records a keypress and appends a row to a CSV is fifty lines and works. A spreadsheet with randomised columns works. Argilla is free and open source if you want annotation management, agreement tracking and a proper interface."}]},{"t":"p","runs":[{"text":"The expensive part is attention, not software. Thirty to ninety seconds per comparison means 400 comparisons is six to ten hours of somebody’s careful work, and their care degrades after about an hour. Spread it, and re-check agreement on a few items from the start of the session against the end."}]},{"t":"h2","id":"what-it-cannot-tell-you","runs":[{"text":"What it cannot tell you"}]},{"t":"p","runs":[{"text":"A win rate says which answer people prefer when reading two side by side. That is not the same as which answer serves them better in use. People prefer confident answers to hedged ones, complete-looking ones to short ones, and agreement to correction, and in every one of those cases the preferred answer is sometimes the worse one."}]},{"t":"p","runs":[{"text":"So pair it with something objective wherever you can: a correctness check, a verifier, a task-completion rate from real use. Preference is one instrument, it measures one thing, and a project that measures only preference will optimise for being liked."}]}],"check":{"question":[{"text":"A tuned model wins 60% of 400 blind pairwise comparisons. Its answers average 148 tokens against the baseline’s 96. What should be checked before accepting the result?"}],"options":[{"text":[{"text":"Whether the judges were consistent, by recomputing the win rate per judge and discarding outliers."}],"correct":false,"tempting":[{"text":"Proposes a reasonable consistency check, but discarding judges who disagree removes exactly the signal that agreement rates are meant to expose."}]},{"text":[{"text":"Whether 400 comparisons is enough, since a 60% win rate needs about 1,500 to exclude 50%."}],"correct":false,"tempting":[{"text":"Misapplies the sample-size guidance; 400 is ample for a 60% rate, and 1,500 is what a 55% rate would require."}]},{"text":[{"text":"Whether the win rate holds on length-matched items, since length buys preference on its own."}],"correct":true,"tempting":null},{"text":[{"text":"Whether the tie rate was high, since ties reduce the effective sample and widen the interval."}],"correct":false,"tempting":[{"text":"Names a genuine effect on precision that does not threaten the validity of the comparison the way an uncontrolled length difference does."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"a-held-out-set-that-holds","title":"A held-out set that actually holds","minutes":9},"next":{"slug":"judging-a-model-you-distilled","title":"Judging a model you distilled from the judge","minutes":9}}}