Turning pairwise wins into a ranking you can defend

Why pairwise, and what it costs

People and judges are far more consistent asked “which of these two is better” than asked “rate this out of ten”. Relative judgements do not drift over a session the way absolute scales do, and they do not require everyone to share a private notion of what 7 means.

The cost is that a pile of pairwise outcomes is not a ranking, and turning it into one is where teams either do something principled or invent a number.

Protocol first, because the aggregation cannot repair a bad protocol: randomise which candidate is shown first, run every pair in both orders, and count a win only where both orders agree. The rate at which the two orders disagree is your noise floor, and no margin smaller than it means anything.

Allow a tie as an explicit option. Forcing a choice between two equally good answers manufactures data that is pure noise, and ties are informative: a system that ties with a much cheaper model on 60% of items is telling you something about whether the expensive one is worth running.

From wins to scores: Bradley-Terry

The standard model says each system has a strength s, and:

P(A beats B) = 1 / (1 + exp(-(s_A - s_B)))

Fit it by logistic regression over your comparison records. One row per comparison, one column per system, +1 for the left system, -1 for the right, target 1 if left won. The fitted coefficients are the strengths, and they come with standard errors for free.

import numpy as np
from sklearn.linear_model import LogisticRegression
X = np.zeros((len(comparisons), n_systems))
for i, (a, b, left_won) in enumerate(comparisons):
    X[i, a], X[i, b] = 1, -1
LogisticRegression(fit_intercept=False, C=1e6).fit(X, y).coef_[0]

Elo is an online approximation of the same model. It updates after each match by a fixed step, which makes it order-dependent and sensitive to the step size: two properties you do not want in an evaluation. Elo exists because chess needed a rating that could be updated by hand between tournaments. If you have all the comparisons in a file, fit the model properly; use Elo only where results arrive continuously and you need a live number.

If you do quote Elo-style numbers, know the conversion: a gap of 100 points corresponds to a win probability of 1 / (1 + 10^(-100/400)), about 64%. A gap of 30 points is a 54% win rate, which is close to nothing.

Put an interval on the rank

Bootstrap over comparisons (resample the comparison records with replacement, refit, record each system’s rank) and report the interval. The usual result on a leaderboard of six systems is that ranks 2 to 4 are statistically indistinguishable and the ordering between them changes on every resample.

Reporting “second, third and fourth are tied within the data we have” is more useful than an ordered list, and it stops a team spending a month chasing a position that does not exist.

How many comparisons

For three to six systems, run every pair; it is cheap and it avoids any question about the design. Above that, all-pairs grows quadratically and adaptive schemes help: compare systems whose current intervals overlap, and stop comparing pairs that have separated. Roughly k log k well-chosen comparisons per round is enough to order k systems whose true gaps are not tiny.

The thing preference does not measure

Preference is not accuracy. Human raters and judge models both reward length, structure, confident phrasing, headings and bullet points. A change that adds a summary paragraph and three bullets will win comparisons while being no more correct, and sometimes less.

So run the two measurements together and read them as a pair: preference win rate, and factual accuracy or task success on the same items. When preference goes up and accuracy stays flat, you have made the output more likeable, which is a real and legitimate product goal, provided you say that is what you did.

This is also the reason to treat public preference arenas as a shortlist rather than a decision. They aggregate crowd preferences on other people’s prompts, using a population that is not your users, on tasks that are not your task, with the same length-and-confidence bias baked in.