Ranking systems, where the labels are clicks and the clicks are biased

Your labels were produced by the system you are grading

Offline ranking metrics assume you know which results were relevant. In a live product you do not; you have clicks. And a click is caused by two things at once: whether the item was any good, and whether it was seen at all.

Examination falls steeply with rank. In typical web results the item at rank one is examined several times more often than the item at rank five, and items below the fold are examined rarely. So the click-through rate at rank one being higher than at rank five tells you almost nothing about relevance, and a model trained or evaluated on raw clicks will learn to reproduce whatever the previous ranker put on top.

Two ways out

Estimate the examination curve and divide it out. Occasionally swap adjacent pairs at random, or randomise the top-k order for a small slice of traffic. Comparing click rates for the same item at different positions gives you the examination probability per rank. Then weight each observed click by the inverse of that probability (inverse propensity scoring) so a click at rank 8 counts far more than one at rank 1.

The catch is variance: dividing by a small probability produces enormous weights, so clip the propensities (a floor around 0.05 is common) and accept a small bias in exchange for a usable estimate.

Interleave instead of splitting. Rather than showing ranker A to half the users and B to the other half, merge both rankings into a single list (take alternate picks from each, tracking whose pick each item was) and attribute clicks to the ranker that contributed the item.

Every user now compares both systems, so the between-user variance disappears from the comparison. Interleaving typically needs one to two orders of magnitude less traffic than an A/B test for the same sensitivity, which turns a two-week experiment into a day. It only works for ranked or multi-option outputs, and it measures preference between rankings rather than any absolute quality.

Offline replay, and the policy you cannot evaluate

You have logs of what was shown and what was clicked, and a new ranker. The temptation is to replay: score the logged items with the new model, see whether it ranks the clicked ones higher.

This works only for items the old system showed. A ranker whose whole point is to surface things the old one never displayed has no logged feedback for exactly its most important behaviour. Off-policy estimators (inverse propensity, doubly robust) extend the reach a little by weighting logged outcomes towards the new policy’s choices, but their variance grows as the two policies diverge, and past a certain distance the estimate is unusable.

The honest statement is a limit: “we can estimate this ranker offline because it overlaps 70% with what was shown; the reordered tail is not evaluable and needs live traffic”.

Metrics beyond relevance

Ranking quality alone will steer you into a system that recommends the same twenty popular items forever, and that is a real business failure rather than a theoretical one. Track alongside your relevance metric:

• Catalogue coverage: the share of items shown to anyone in a week.
• Concentration: a Gini coefficient over impressions, watched for drift over months.
• Intra-list diversity: average pairwise similarity of the items in one result page.
• Novelty: share of recommendations the user has not already seen.

The feedback loop is the mechanism to keep in mind: popular items get shown, shown items get clicked, clicked items are treated as good, and the model gets more confident about a narrowing set. Nothing in a relevance metric detects this, because within the loop the model is right.

Slices that decide the product

• New users with no history, and new items with no interactions. Both are where recommenders fail hardest and both are usually invisible in an overall number, since existing users and popular items dominate the traffic.
• Rare queries. Half of search traffic is often queries seen a handful of times; measure the tail separately, and note that keyword matching frequently beats embeddings there because rare exact terms are exactly what an average vector loses.
• Zero-result and one-result queries, which are pure failures the average will not show you.

Free tools cover the offline side: pytrec_eval for standard ranking metrics, rank_bm25 for the keyword baseline, implicit for matrix-factorisation baselines. The online side needs no library, only the discipline to randomise something.