{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"when-the-eval-set-goes-stale","title":"Refreshing a set without breaking every comparison","url":"https://addaly.com/learn/evaluating-ai/when-the-eval-set-goes-stale","position":72,"of":76,"minutes":8,"answer":"Compare your set's mix against last month's traffic and retire it when the divergence gets large, but change it on a schedule with both versions reported for one cycle.","keyIdea":"Compare your set's mix against last month's traffic and retire it when the divergence gets large, but change it on a schedule with both versions reported for one cycle, because a silent set change looks exactly like a quality change."},"body":{"format":"markdown","text":"## The symptom\n\nThe evaluation score has been steady at 87% for eight months. Complaints are rising. Nothing in the harness is broken.\n\nThe set was drawn in March. Since then a new market opened, the app shipped a feature that changed how people phrase requests, and a large customer started sending documents rather than questions. The set is measuring a product that no longer exists, precisely and reproducibly.\n\n## Measure the divergence, do not guess it\n\nYou do not need a sophisticated drift detector. Bucket both your eval set and last month's traffic on a few cheap features, and compare the proportions:\n\n- intent or topic mix\n- language and script\n- input length, in three or four bands\n- channel or surface\n- whether an attachment or a document is present\n\nThen sum the absolute differences and halve it, which gives the total variation distance between the two mixes:\n\n```python\ntvd = 0.5 * sum(abs(p_eval[b] - p_live[b]) for b in buckets)\n```\n\nA value under 0.1 means the set still resembles traffic. Around 0.2 the headline number is drifting away from reality. Above 0.3 you are reporting a number about a different population, and the refresh is overdue.\n\nRun this monthly. It takes a query and a few lines, and it converts \"the set feels old\" into a scheduled decision.\n\n## Refresh on a schedule, in a way that preserves comparisons\n\nChanging a set changes the number, and a number that moves because the set moved looks exactly like a quality change. So:\n\n- **Version the set**, with a date, and record the version alongside every result. A number with no set version cannot be compared with anything.\n- **Keep a frozen core** (perhaps half the items) unchanged across refreshes, so there is always a like-for-like comparison across the boundary.\n- **Report both versions for one cycle.** Old set 87%, new set 81%, both on the same system, and the six-point gap is a fact about the sets rather than a regression. Publish that gap; it is the most informative number in the whole refresh.\n- **Never swap silently.** Every incident where a team argued for a week about a phantom regression traces back to somebody quietly adding forty items.\n\n## Retire items, without losing what they guard\n\nAn item that every version has passed for a year carries no information. It costs money and it never moves. But deleting it removes the guard against a regression that was fixed long ago, and fixed bugs do come back.\n\nMove rather than delete. Retired items go to a cheap smoke suite that runs nightly with deterministic checks only, or in a cascade's lowest tier. The regression is still guarded and the expensive judging budget goes to items that discriminate.\n\nThe reverse rule matters too: an item that flips constantly is either a rubric ambiguity or genuine non-determinism, and it belongs in quarantine rather than in the headline set.\n\n## Keep the composition honest\n\nThere is a specific way sets rot even when they are maintained. Every incident adds an item, so after two years the set is a museum of past bugs; and each of those items was chosen precisely because the system failed it, which makes the set systematically unrepresentative and systematically hard.\n\nHold a ratio. Something like: no more than a third of the set from incidents, the rest sampled from traffic on the refresh schedule. When incidents push past the ratio, either sample more traffic or move the older incident items into the regression suite, which is where they belonged all along.\n\n## The labels go stale too\n\nA set can match today's traffic perfectly and still be wrong, because the answers have moved. The refund window changed, a product was discontinued, a policy was rewritten, a fee went up. Every reference answer touching those topics is now incorrect, and the evaluation is quietly penalising the right behaviour.\n\nTwo defences. Attach a policy version to the items that depend on one, as the reference-answers lesson described, and fail the run loudly when the versions diverge. And when a regression appears on a cluster of related items at once (five refund questions failing on the same day) check the policy before you check the model. Roughly half the time the model is right and the file is old.\n\n## Give it an owner and a date\n\nSets rot when nobody owns them. Put a name and a review date in the set's metadata file, review quarterly, and record what changed and why. It is fifteen minutes a quarter, and it is the difference between a measurement system and a folder of old JSON that everybody has stopped believing."},"check":{"question":"A team replaces its evaluation set and the reported score drops from 87% to 81% with no change to the system. What should they publish?","options":[{"text":"The new 81%, since the new set better represents current traffic and the old figure is obsolete.","correct":false,"tempting":"Publishes the right number with no way for a reader to reconcile it against every previous report, which is how a set change gets read as a regression."},{"text":"Both figures on the same system for one cycle, so the six-point gap is visible as a difference between sets rather than a change in quality.","correct":true},{"text":"The old 87% until enough history accumulates on the new set to make it comparable.","correct":false,"tempting":"Continues reporting a number about a population that no longer exists, which is the problem the refresh was meant to fix."},{"text":"An average of the two, weighted by set size, to smooth the transition.","correct":false,"tempting":"Blends two measurements of different populations into a figure that describes neither and cannot be reproduced by anyone."}]},"prev":{"slug":"feedback-loops-and-self-poisoning","title":"The system that trains on its own output","url":"https://addaly.com/learn/evaluating-ai/feedback-loops-and-self-poisoning"},"next":{"slug":"fairness-you-can-measure","title":"Fairness as arithmetic, and the choice you cannot avoid","url":"https://addaly.com/learn/evaluating-ai/fairness-you-can-measure"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Refreshing a set without breaking every comparison', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/when-the-eval-set-goes-stale (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Refreshing a set without breaking every comparison","courseTitle":"Knowing If It Works","module":{"index":8,"slug":"living-with-it","title":"Living with it"},"answer":null,"keyIdea":"Compare your set’s mix against last month’s traffic and retire it when the divergence gets large, but change it on a schedule with both versions reported for one cycle, because a silent set change looks exactly like a quality change.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-symptom","runs":[{"text":"The symptom"}]},{"t":"p","runs":[{"text":"The evaluation score has been steady at 87% for eight months. Complaints are rising. Nothing in the harness is broken."}]},{"t":"p","runs":[{"text":"The set was drawn in March. Since then a new market opened, the app shipped a feature that changed how people phrase requests, and a large customer started sending documents rather than questions. The set is measuring a product that no longer exists, precisely and reproducibly."}]},{"t":"h2","id":"measure-the-divergence-do-not-guess-it","runs":[{"text":"Measure the divergence, do not guess it"}]},{"t":"p","runs":[{"text":"You do not need a sophisticated drift detector. Bucket both your eval set and last month’s traffic on a few cheap features, and compare the proportions:"}]},{"t":"ul","items":[[{"text":"intent or topic mix"}],[{"text":"language and script"}],[{"text":"input length, in three or four bands"}],[{"text":"channel or surface"}],[{"text":"whether an attachment or a document is present"}]]},{"t":"p","runs":[{"text":"Then sum the absolute differences and halve it, which gives the total variation distance between the two mixes:"}]},{"t":"code","lang":"python","label":"Python","text":"tvd = 0.5 * sum(abs(p_eval[b] - p_live[b]) for b in buckets)"},{"t":"p","runs":[{"text":"A value under 0.1 means the set still resembles traffic. Around 0.2 the headline number is drifting away from reality. Above 0.3 you are reporting a number about a different population, and the refresh is overdue."}]},{"t":"p","runs":[{"text":"Run this monthly. It takes a query and a few lines, and it converts “the set feels old” into a scheduled decision."}]},{"t":"h2","id":"refresh-on-a-schedule-in-a-way-that-preserves-comparisons","runs":[{"text":"Refresh on a schedule, in a way that preserves comparisons"}]},{"t":"p","runs":[{"text":"Changing a set changes the number, and a number that moves because the set moved looks exactly like a quality change. So:"}]},{"t":"ul","items":[[{"text":"Version the set","b":true},{"text":", with a date, and record the version alongside every result. A number with no set version cannot be compared with anything."}],[{"text":"Keep a frozen core","b":true},{"text":" (perhaps half the items) unchanged across refreshes, so there is always a like-for-like comparison across the boundary."}],[{"text":"Report both versions for one cycle.","b":true},{"text":" Old set 87%, new set 81%, both on the same system, and the six-point gap is a fact about the sets rather than a regression. Publish that gap; it is the most informative number in the whole refresh."}],[{"text":"Never swap silently.","b":true},{"text":" Every incident where a team argued for a week about a phantom regression traces back to somebody quietly adding forty items."}]]},{"t":"h2","id":"retire-items-without-losing-what-they-guard","runs":[{"text":"Retire items, without losing what they guard"}]},{"t":"p","runs":[{"text":"An item that every version has passed for a year carries no information. It costs money and it never moves. But deleting it removes the guard against a regression that was fixed long ago, and fixed bugs do come back."}]},{"t":"p","runs":[{"text":"Move rather than delete. Retired items go to a cheap smoke suite that runs nightly with deterministic checks only, or in a cascade’s lowest tier. The regression is still guarded and the expensive judging budget goes to items that discriminate."}]},{"t":"p","runs":[{"text":"The reverse rule matters too: an item that flips constantly is either a rubric ambiguity or genuine non-determinism, and it belongs in quarantine rather than in the headline set."}]},{"t":"h2","id":"keep-the-composition-honest","runs":[{"text":"Keep the composition honest"}]},{"t":"p","runs":[{"text":"There is a specific way sets rot even when they are maintained. Every incident adds an item, so after two years the set is a museum of past bugs; and each of those items was chosen precisely because the system failed it, which makes the set systematically unrepresentative and systematically hard."}]},{"t":"p","runs":[{"text":"Hold a ratio. Something like: no more than a third of the set from incidents, the rest sampled from traffic on the refresh schedule. When incidents push past the ratio, either sample more traffic or move the older incident items into the regression suite, which is where they belonged all along."}]},{"t":"h2","id":"the-labels-go-stale-too","runs":[{"text":"The labels go stale too"}]},{"t":"p","runs":[{"text":"A set can match today’s traffic perfectly and still be wrong, because the answers have moved. The refund window changed, a product was discontinued, a policy was rewritten, a fee went up. Every reference answer touching those topics is now incorrect, and the evaluation is quietly penalising the right behaviour."}]},{"t":"p","runs":[{"text":"Two defences. Attach a policy version to the items that depend on one, as the reference-answers lesson described, and fail the run loudly when the versions diverge. And when a regression appears on a cluster of related items at once (five refund questions failing on the same day) check the policy before you check the model. Roughly half the time the model is right and the file is old."}]},{"t":"h2","id":"give-it-an-owner-and-a-date","runs":[{"text":"Give it an owner and a date"}]},{"t":"p","runs":[{"text":"Sets rot when nobody owns them. Put a name and a review date in the set’s metadata file, review quarterly, and record what changed and why. It is fifteen minutes a quarter, and it is the difference between a measurement system and a folder of old JSON that everybody has stopped believing."}]}],"check":{"question":[{"text":"A team replaces its evaluation set and the reported score drops from 87% to 81% with no change to the system. What should they publish?"}],"options":[{"text":[{"text":"The new 81%, since the new set better represents current traffic and the old figure is obsolete."}],"correct":false,"tempting":[{"text":"Publishes the right number with no way for a reader to reconcile it against every previous report, which is how a set change gets read as a regression."}]},{"text":[{"text":"Both figures on the same system for one cycle, so the six-point gap is visible as a difference between sets rather than a change in quality."}],"correct":true,"tempting":null},{"text":[{"text":"The old 87% until enough history accumulates on the new set to make it comparable."}],"correct":false,"tempting":[{"text":"Continues reporting a number about a population that no longer exists, which is the problem the refresh was meant to fix."}]},{"text":[{"text":"An average of the two, weighted by set size, to smooth the transition."}],"correct":false,"tempting":[{"text":"Blends two measurements of different populations into a figure that describes neither and cannot be reproduced by anyone."}]}],"order":[0,3,2,1],"why":null},"prev":{"slug":"feedback-loops-and-self-poisoning","title":"The system that trains on its own output","minutes":9},"next":{"slug":"fairness-you-can-measure","title":"Fairness as arithmetic, and the choice you cannot avoid","minutes":10}}}