{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"ranking-systems-by-preference","title":"Turning pairwise wins into a ranking you can defend","url":"https://addaly.com/learn/evaluating-ai/ranking-systems-by-preference","position":44,"of":76,"minutes":9,"answer":"Fit a Bradley-Terry model to your pairwise comparisons rather than accumulating Elo, put a bootstrap interval on every rank, and remember that preference rewards length and confidence, so measure accuracy separately.","keyIdea":"Fit a Bradley-Terry model to your pairwise comparisons rather than accumulating Elo, put a bootstrap interval on every rank, and remember that preference rewards length and confidence, so measure accuracy separately."},"body":{"format":"markdown","text":"## Why pairwise, and what it costs\n\nPeople and judges are far more consistent asked \"which of these two is better\" than asked \"rate this out of ten\". Relative judgements do not drift over a session the way absolute scales do, and they do not require everyone to share a private notion of what 7 means.\n\nThe cost is that a pile of pairwise outcomes is not a ranking, and turning it into one is where teams either do something principled or invent a number.\n\nProtocol first, because the aggregation cannot repair a bad protocol: randomise which candidate is shown first, run every pair in both orders, and count a win only where both orders agree. The rate at which the two orders disagree is your noise floor, and no margin smaller than it means anything.\n\nAllow a **tie** as an explicit option. Forcing a choice between two equally good answers manufactures data that is pure noise, and ties are informative: a system that ties with a much cheaper model on 60% of items is telling you something about whether the expensive one is worth running.\n\n## From wins to scores: Bradley-Terry\n\nThe standard model says each system has a strength `s`, and:\n\n```\nP(A beats B) = 1 / (1 + exp(-(s_A - s_B)))\n```\n\nFit it by logistic regression over your comparison records. One row per comparison, one column per system, `+1` for the left system, `-1` for the right, target 1 if left won. The fitted coefficients are the strengths, and they come with standard errors for free.\n\n```python\nimport numpy as np\nfrom sklearn.linear_model import LogisticRegression\nX = np.zeros((len(comparisons), n_systems))\nfor i, (a, b, left_won) in enumerate(comparisons):\n    X[i, a], X[i, b] = 1, -1\nLogisticRegression(fit_intercept=False, C=1e6).fit(X, y).coef_[0]\n```\n\n**Elo is an online approximation of the same model.** It updates after each match by a fixed step, which makes it order-dependent and sensitive to the step size: two properties you do not want in an evaluation. Elo exists because chess needed a rating that could be updated by hand between tournaments. If you have all the comparisons in a file, fit the model properly; use Elo only where results arrive continuously and you need a live number.\n\nIf you do quote Elo-style numbers, know the conversion: a gap of 100 points corresponds to a win probability of `1 / (1 + 10^(-100/400))`, about 64%. A gap of 30 points is a 54% win rate, which is close to nothing.\n\n## Put an interval on the rank\n\nBootstrap over comparisons (resample the comparison records with replacement, refit, record each system's rank) and report the interval. The usual result on a leaderboard of six systems is that ranks 2 to 4 are statistically indistinguishable and the ordering between them changes on every resample.\n\nReporting \"second, third and fourth are tied within the data we have\" is more useful than an ordered list, and it stops a team spending a month chasing a position that does not exist.\n\n## How many comparisons\n\nFor three to six systems, run every pair; it is cheap and it avoids any question about the design. Above that, all-pairs grows quadratically and adaptive schemes help: compare systems whose current intervals overlap, and stop comparing pairs that have separated. Roughly `k log k` well-chosen comparisons per round is enough to order `k` systems whose true gaps are not tiny.\n\n## The thing preference does not measure\n\nPreference is not accuracy. Human raters and judge models both reward length, structure, confident phrasing, headings and bullet points. A change that adds a summary paragraph and three bullets will win comparisons while being no more correct, and sometimes less.\n\nSo run the two measurements together and read them as a pair: preference win rate, and factual accuracy or task success on the same items. When preference goes up and accuracy stays flat, you have made the output more likeable, which is a real and legitimate product goal, provided you say that is what you did.\n\nThis is also the reason to treat public preference arenas as a shortlist rather than a decision. They aggregate crowd preferences on other people's prompts, using a population that is not your users, on tasks that are not your task, with the same length-and-confidence bias baked in."},"check":{"question":"A leaderboard built from pairwise judging shows model B ahead of model C by 25 Elo-equivalent points after 400 comparisons. What is the defensible reading?","options":[{"text":"B is meaningfully better and should be adopted, since the comparison used a proper pairwise protocol.","correct":false,"tempting":"Treats a small gap from a proper protocol as decisive; 25 points is roughly a 54% win rate, which 400 comparisons cannot separate from a tie."},{"text":"B and C are close to tied: 25 points implies about a 54% win rate, and a bootstrap over the comparisons will very likely show overlapping ranks.","correct":true},{"text":"The gap cannot be interpreted at all without knowing which model was shown first.","correct":false,"tempting":"Order matters and is handled by running both orders; once that is done, the gap is interpretable, just small."},{"text":"B is better on style but the ranking says nothing about which is more accurate, so the comparison is meaningless.","correct":false,"tempting":"Correctly notes that preference is not accuracy, then overstates it: the ranking is meaningful about preference, and the problem here is the size of the gap."}]},"prev":{"slug":"calibrating-and-monitoring-the-judge","title":"Proving the judge agrees with you, and re-proving it","url":"https://addaly.com/learn/evaluating-ai/calibrating-and-monitoring-the-judge"},"next":{"slug":"cheap-checks-first","title":"The checks that cost nothing and catch most of it","url":"https://addaly.com/learn/evaluating-ai/cheap-checks-first"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Turning pairwise wins into a ranking you can defend', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/ranking-systems-by-preference (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Turning pairwise wins into a ranking you can defend","courseTitle":"Knowing If It Works","module":{"index":5,"slug":"automating-judgement","title":"Automating judgement"},"answer":null,"keyIdea":"Fit a Bradley-Terry model to your pairwise comparisons rather than accumulating Elo, put a bootstrap interval on every rank, and remember that preference rewards length and confidence, so measure accuracy separately.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-pairwise-and-what-it-costs","runs":[{"text":"Why pairwise, and what it costs"}]},{"t":"p","runs":[{"text":"People and judges are far more consistent asked “which of these two is better” than asked “rate this out of ten”. Relative judgements do not drift over a session the way absolute scales do, and they do not require everyone to share a private notion of what 7 means."}]},{"t":"p","runs":[{"text":"The cost is that a pile of pairwise outcomes is not a ranking, and turning it into one is where teams either do something principled or invent a number."}]},{"t":"p","runs":[{"text":"Protocol first, because the aggregation cannot repair a bad protocol: randomise which candidate is shown first, run every pair in both orders, and count a win only where both orders agree. The rate at which the two orders disagree is your noise floor, and no margin smaller than it means anything."}]},{"t":"p","runs":[{"text":"Allow a "},{"text":"tie","b":true},{"text":" as an explicit option. Forcing a choice between two equally good answers manufactures data that is pure noise, and ties are informative: a system that ties with a much cheaper model on 60% of items is telling you something about whether the expensive one is worth running."}]},{"t":"h2","id":"from-wins-to-scores-bradley-terry","runs":[{"text":"From wins to scores: Bradley-Terry"}]},{"t":"p","runs":[{"text":"The standard model says each system has a strength "},{"text":"s","code":true},{"text":", and:"}]},{"t":"code","lang":"","label":"Text","text":"P(A beats B) = 1 / (1 + exp(-(s_A - s_B)))"},{"t":"p","runs":[{"text":"Fit it by logistic regression over your comparison records. One row per comparison, one column per system, "},{"text":"+1","code":true},{"text":" for the left system, "},{"text":"-1","code":true},{"text":" for the right, target 1 if left won. The fitted coefficients are the strengths, and they come with standard errors for free."}]},{"t":"code","lang":"python","label":"Python","text":"import numpy as np\nfrom sklearn.linear_model import LogisticRegression\nX = np.zeros((len(comparisons), n_systems))\nfor i, (a, b, left_won) in enumerate(comparisons):\n    X[i, a], X[i, b] = 1, -1\nLogisticRegression(fit_intercept=False, C=1e6).fit(X, y).coef_[0]"},{"t":"p","runs":[{"text":"Elo is an online approximation of the same model.","b":true},{"text":" It updates after each match by a fixed step, which makes it order-dependent and sensitive to the step size: two properties you do not want in an evaluation. Elo exists because chess needed a rating that could be updated by hand between tournaments. If you have all the comparisons in a file, fit the model properly; use Elo only where results arrive continuously and you need a live number."}]},{"t":"p","runs":[{"text":"If you do quote Elo-style numbers, know the conversion: a gap of 100 points corresponds to a win probability of "},{"text":"1 / (1 + 10^(-100/400))","code":true},{"text":", about 64%. A gap of 30 points is a 54% win rate, which is close to nothing."}]},{"t":"h2","id":"put-an-interval-on-the-rank","runs":[{"text":"Put an interval on the rank"}]},{"t":"p","runs":[{"text":"Bootstrap over comparisons (resample the comparison records with replacement, refit, record each system’s rank) and report the interval. The usual result on a leaderboard of six systems is that ranks 2 to 4 are statistically indistinguishable and the ordering between them changes on every resample."}]},{"t":"p","runs":[{"text":"Reporting “second, third and fourth are tied within the data we have” is more useful than an ordered list, and it stops a team spending a month chasing a position that does not exist."}]},{"t":"h2","id":"how-many-comparisons","runs":[{"text":"How many comparisons"}]},{"t":"p","runs":[{"text":"For three to six systems, run every pair; it is cheap and it avoids any question about the design. Above that, all-pairs grows quadratically and adaptive schemes help: compare systems whose current intervals overlap, and stop comparing pairs that have separated. Roughly "},{"text":"k log k","code":true},{"text":" well-chosen comparisons per round is enough to order "},{"text":"k","code":true},{"text":" systems whose true gaps are not tiny."}]},{"t":"h2","id":"the-thing-preference-does-not-measure","runs":[{"text":"The thing preference does not measure"}]},{"t":"p","runs":[{"text":"Preference is not accuracy. Human raters and judge models both reward length, structure, confident phrasing, headings and bullet points. A change that adds a summary paragraph and three bullets will win comparisons while being no more correct, and sometimes less."}]},{"t":"p","runs":[{"text":"So run the two measurements together and read them as a pair: preference win rate, and factual accuracy or task success on the same items. When preference goes up and accuracy stays flat, you have made the output more likeable, which is a real and legitimate product goal, provided you say that is what you did."}]},{"t":"p","runs":[{"text":"This is also the reason to treat public preference arenas as a shortlist rather than a decision. They aggregate crowd preferences on other people’s prompts, using a population that is not your users, on tasks that are not your task, with the same length-and-confidence bias baked in."}]}],"check":{"question":[{"text":"A leaderboard built from pairwise judging shows model B ahead of model C by 25 Elo-equivalent points after 400 comparisons. What is the defensible reading?"}],"options":[{"text":[{"text":"B is meaningfully better and should be adopted, since the comparison used a proper pairwise protocol."}],"correct":false,"tempting":[{"text":"Treats a small gap from a proper protocol as decisive; 25 points is roughly a 54% win rate, which 400 comparisons cannot separate from a tie."}]},{"text":[{"text":"B and C are close to tied: 25 points implies about a 54% win rate, and a bootstrap over the comparisons will very likely show overlapping ranks."}],"correct":true,"tempting":null},{"text":[{"text":"The gap cannot be interpreted at all without knowing which model was shown first."}],"correct":false,"tempting":[{"text":"Order matters and is handled by running both orders; once that is done, the gap is interpretable, just small."}]},{"text":[{"text":"B is better on style but the ranking says nothing about which is more accurate, so the comparison is meaningless."}],"correct":false,"tempting":[{"text":"Correctly notes that preference is not accuracy, then overstates it: the ranking is meaningful about preference, and the problem here is the size of the gap."}]}],"order":[2,3,1,0],"why":null},"prev":{"slug":"calibrating-and-monitoring-the-judge","title":"Proving the judge agrees with you, and re-proving it","minutes":9},"next":{"slug":"cheap-checks-first","title":"The checks that cost nothing and catch most of it","minutes":8}}}