{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"evaluating-search-and-recommendation","title":"Ranking systems, where the labels are clicks and the clicks are biased","url":"https://addaly.com/learn/evaluating-ai/evaluating-search-and-recommendation","position":35,"of":76,"minutes":10,"answer":"A click measures position as much as relevance, so correct for examination bias, compare rankers by interleaving rather than by A/B, and watch catalogue coverage because a recommender teaches itself what to show next.","keyIdea":"A click measures position as much as relevance, so correct for examination bias, compare rankers by interleaving rather than by A/B, and watch catalogue coverage because a recommender teaches itself what to show next."},"body":{"format":"markdown","text":"## Your labels were produced by the system you are grading\n\nOffline ranking metrics assume you know which results were relevant. In a live product you do not; you have clicks. And a click is caused by two things at once: whether the item was any good, and whether it was seen at all.\n\nExamination falls steeply with rank. In typical web results the item at rank one is examined several times more often than the item at rank five, and items below the fold are examined rarely. So the click-through rate at rank one being higher than at rank five tells you almost nothing about relevance, and a model trained or evaluated on raw clicks will learn to reproduce whatever the previous ranker put on top.\n\n## Two ways out\n\n**Estimate the examination curve and divide it out.** Occasionally swap adjacent pairs at random, or randomise the top-k order for a small slice of traffic. Comparing click rates for the same item at different positions gives you the examination probability per rank. Then weight each observed click by the inverse of that probability (inverse propensity scoring) so a click at rank 8 counts far more than one at rank 1.\n\nThe catch is variance: dividing by a small probability produces enormous weights, so clip the propensities (a floor around 0.05 is common) and accept a small bias in exchange for a usable estimate.\n\n**Interleave instead of splitting.** Rather than showing ranker A to half the users and B to the other half, merge both rankings into a single list (take alternate picks from each, tracking whose pick each item was) and attribute clicks to the ranker that contributed the item.\n\nEvery user now compares both systems, so the between-user variance disappears from the comparison. Interleaving typically needs one to two orders of magnitude less traffic than an A/B test for the same sensitivity, which turns a two-week experiment into a day. It only works for ranked or multi-option outputs, and it measures preference between rankings rather than any absolute quality.\n\n## Offline replay, and the policy you cannot evaluate\n\nYou have logs of what was shown and what was clicked, and a new ranker. The temptation is to replay: score the logged items with the new model, see whether it ranks the clicked ones higher.\n\nThis works only for items the old system showed. A ranker whose whole point is to surface things the old one never displayed has no logged feedback for exactly its most important behaviour. Off-policy estimators (inverse propensity, doubly robust) extend the reach a little by weighting logged outcomes towards the new policy's choices, but their variance grows as the two policies diverge, and past a certain distance the estimate is unusable.\n\nThe honest statement is a limit: \"we can estimate this ranker offline because it overlaps 70% with what was shown; the reordered tail is not evaluable and needs live traffic\".\n\n## Metrics beyond relevance\n\nRanking quality alone will steer you into a system that recommends the same twenty popular items forever, and that is a real business failure rather than a theoretical one. Track alongside your relevance metric:\n\n- **Catalogue coverage**: the share of items shown to anyone in a week.\n- **Concentration**: a Gini coefficient over impressions, watched for drift over months.\n- **Intra-list diversity**: average pairwise similarity of the items in one result page.\n- **Novelty**: share of recommendations the user has not already seen.\n\nThe feedback loop is the mechanism to keep in mind: popular items get shown, shown items get clicked, clicked items are treated as good, and the model gets more confident about a narrowing set. Nothing in a relevance metric detects this, because within the loop the model is right.\n\n## Slices that decide the product\n\n- **New users** with no history, and **new items** with no interactions. Both are where recommenders fail hardest and both are usually invisible in an overall number, since existing users and popular items dominate the traffic.\n- **Rare queries.** Half of search traffic is often queries seen a handful of times; measure the tail separately, and note that keyword matching frequently beats embeddings there because rare exact terms are exactly what an average vector loses.\n- **Zero-result and one-result queries**, which are pure failures the average will not show you.\n\nFree tools cover the offline side: `pytrec_eval` for standard ranking metrics, `rank_bm25` for the keyword baseline, `implicit` for matrix-factorisation baselines. The online side needs no library, only the discipline to randomise something."},"check":{"question":"A team compares two rankers by measuring click-through rate on the top result for each. Why is that comparison weak evidence about relevance?","options":[{"text":"Click-through rate is a proxy metric, and proxy metrics must always be validated against human labels before use.","correct":false,"tempting":"True as a general principle and not the specific mechanism; even a validated proxy measured this way inherits the position effect."},{"text":"A click at rank one reflects that the item was examined because it was at rank one, so the measurement mixes examination probability with relevance and favours whatever the previous ranker promoted.","correct":true},{"text":"Click-through rate ignores dwell time, so a click on a bad result counts the same as a click on a good one.","correct":false,"tempting":"A genuine weakness of raw clicks that argues for a better outcome signal, but it does not explain the systematic bias created by position."},{"text":"Top-result click-through rate cannot be compared across rankers because the two rankers show different items.","correct":false,"tempting":"Showing different items is the point of the comparison; the flaw is the position-driven examination bias, not the difference in items."}]},"prev":{"slug":"evaluating-conversation","title":"Multi-turn: score the session, not the turn","url":"https://addaly.com/learn/evaluating-ai/evaluating-conversation"},"next":{"slug":"evaluating-vision-models","title":"Images: the background the model is actually reading","url":"https://addaly.com/learn/evaluating-ai/evaluating-vision-models"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Ranking systems, where the labels are clicks and the clicks are biased', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/evaluating-search-and-recommendation (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Ranking systems, where the labels are clicks and the clicks are biased","courseTitle":"Knowing If It Works","module":{"index":4,"slug":"task-shapes","title":"The systems you will actually be handed"},"answer":null,"keyIdea":"A click measures position as much as relevance, so correct for examination bias, compare rankers by interleaving rather than by A/B, and watch catalogue coverage because a recommender teaches itself what to show next.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"your-labels-were-produced-by-the-system-you-are-grading","runs":[{"text":"Your labels were produced by the system you are grading"}]},{"t":"p","runs":[{"text":"Offline ranking metrics assume you know which results were relevant. In a live product you do not; you have clicks. And a click is caused by two things at once: whether the item was any good, and whether it was seen at all."}]},{"t":"p","runs":[{"text":"Examination falls steeply with rank. In typical web results the item at rank one is examined several times more often than the item at rank five, and items below the fold are examined rarely. So the click-through rate at rank one being higher than at rank five tells you almost nothing about relevance, and a model trained or evaluated on raw clicks will learn to reproduce whatever the previous ranker put on top."}]},{"t":"h2","id":"two-ways-out","runs":[{"text":"Two ways out"}]},{"t":"p","runs":[{"text":"Estimate the examination curve and divide it out.","b":true},{"text":" Occasionally swap adjacent pairs at random, or randomise the top-k order for a small slice of traffic. Comparing click rates for the same item at different positions gives you the examination probability per rank. Then weight each observed click by the inverse of that probability (inverse propensity scoring) so a click at rank 8 counts far more than one at rank 1."}]},{"t":"p","runs":[{"text":"The catch is variance: dividing by a small probability produces enormous weights, so clip the propensities (a floor around 0.05 is common) and accept a small bias in exchange for a usable estimate."}]},{"t":"p","runs":[{"text":"Interleave instead of splitting.","b":true},{"text":" Rather than showing ranker A to half the users and B to the other half, merge both rankings into a single list (take alternate picks from each, tracking whose pick each item was) and attribute clicks to the ranker that contributed the item."}]},{"t":"p","runs":[{"text":"Every user now compares both systems, so the between-user variance disappears from the comparison. Interleaving typically needs one to two orders of magnitude less traffic than an A/B test for the same sensitivity, which turns a two-week experiment into a day. It only works for ranked or multi-option outputs, and it measures preference between rankings rather than any absolute quality."}]},{"t":"h2","id":"offline-replay-and-the-policy-you-cannot-evaluate","runs":[{"text":"Offline replay, and the policy you cannot evaluate"}]},{"t":"p","runs":[{"text":"You have logs of what was shown and what was clicked, and a new ranker. The temptation is to replay: score the logged items with the new model, see whether it ranks the clicked ones higher."}]},{"t":"p","runs":[{"text":"This works only for items the old system showed. A ranker whose whole point is to surface things the old one never displayed has no logged feedback for exactly its most important behaviour. Off-policy estimators (inverse propensity, doubly robust) extend the reach a little by weighting logged outcomes towards the new policy’s choices, but their variance grows as the two policies diverge, and past a certain distance the estimate is unusable."}]},{"t":"p","runs":[{"text":"The honest statement is a limit: “we can estimate this ranker offline because it overlaps 70% with what was shown; the reordered tail is not evaluable and needs live traffic”."}]},{"t":"h2","id":"metrics-beyond-relevance","runs":[{"text":"Metrics beyond relevance"}]},{"t":"p","runs":[{"text":"Ranking quality alone will steer you into a system that recommends the same twenty popular items forever, and that is a real business failure rather than a theoretical one. Track alongside your relevance metric:"}]},{"t":"ul","items":[[{"text":"Catalogue coverage","b":true},{"text":": the share of items shown to anyone in a week."}],[{"text":"Concentration","b":true},{"text":": a Gini coefficient over impressions, watched for drift over months."}],[{"text":"Intra-list diversity","b":true},{"text":": average pairwise similarity of the items in one result page."}],[{"text":"Novelty","b":true},{"text":": share of recommendations the user has not already seen."}]]},{"t":"p","runs":[{"text":"The feedback loop is the mechanism to keep in mind: popular items get shown, shown items get clicked, clicked items are treated as good, and the model gets more confident about a narrowing set. Nothing in a relevance metric detects this, because within the loop the model is right."}]},{"t":"h2","id":"slices-that-decide-the-product","runs":[{"text":"Slices that decide the product"}]},{"t":"ul","items":[[{"text":"New users","b":true},{"text":" with no history, and "},{"text":"new items","b":true},{"text":" with no interactions. Both are where recommenders fail hardest and both are usually invisible in an overall number, since existing users and popular items dominate the traffic."}],[{"text":"Rare queries.","b":true},{"text":" Half of search traffic is often queries seen a handful of times; measure the tail separately, and note that keyword matching frequently beats embeddings there because rare exact terms are exactly what an average vector loses."}],[{"text":"Zero-result and one-result queries","b":true},{"text":", which are pure failures the average will not show you."}]]},{"t":"p","runs":[{"text":"Free tools cover the offline side: "},{"text":"pytrec_eval","code":true},{"text":" for standard ranking metrics, "},{"text":"rank_bm25","code":true},{"text":" for the keyword baseline, "},{"text":"implicit","code":true},{"text":" for matrix-factorisation baselines. The online side needs no library, only the discipline to randomise something."}]}],"check":{"question":[{"text":"A team compares two rankers by measuring click-through rate on the top result for each. Why is that comparison weak evidence about relevance?"}],"options":[{"text":[{"text":"Click-through rate is a proxy metric, and proxy metrics must always be validated against human labels before use."}],"correct":false,"tempting":[{"text":"True as a general principle and not the specific mechanism; even a validated proxy measured this way inherits the position effect."}]},{"text":[{"text":"A click at rank one reflects that the item was examined because it was at rank one, so the measurement mixes examination probability with relevance and favours whatever the previous ranker promoted."}],"correct":true,"tempting":null},{"text":[{"text":"Click-through rate ignores dwell time, so a click on a bad result counts the same as a click on a good one."}],"correct":false,"tempting":[{"text":"A genuine weakness of raw clicks that argues for a better outcome signal, but it does not explain the systematic bias created by position."}]},{"text":[{"text":"Top-result click-through rate cannot be compared across rankers because the two rankers show different items."}],"correct":false,"tempting":[{"text":"Showing different items is the point of the comparison; the flaw is the position-driven examination bias, not the difference in items."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"evaluating-conversation","title":"Multi-turn: score the session, not the turn","minutes":9},"next":{"slug":"evaluating-vision-models","title":"Images: the background the model is actually reading","minutes":9}}}