Refreshing a set without breaking every comparison

The symptom

The evaluation score has been steady at 87% for eight months. Complaints are rising. Nothing in the harness is broken.

The set was drawn in March. Since then a new market opened, the app shipped a feature that changed how people phrase requests, and a large customer started sending documents rather than questions. The set is measuring a product that no longer exists, precisely and reproducibly.

Measure the divergence, do not guess it

You do not need a sophisticated drift detector. Bucket both your eval set and last month’s traffic on a few cheap features, and compare the proportions:

• intent or topic mix
• language and script
• input length, in three or four bands
• channel or surface
• whether an attachment or a document is present

Then sum the absolute differences and halve it, which gives the total variation distance between the two mixes:

tvd = 0.5 * sum(abs(p_eval[b] - p_live[b]) for b in buckets)

A value under 0.1 means the set still resembles traffic. Around 0.2 the headline number is drifting away from reality. Above 0.3 you are reporting a number about a different population, and the refresh is overdue.

Run this monthly. It takes a query and a few lines, and it converts “the set feels old” into a scheduled decision.

Refresh on a schedule, in a way that preserves comparisons

Changing a set changes the number, and a number that moves because the set moved looks exactly like a quality change. So:

• Version the set, with a date, and record the version alongside every result. A number with no set version cannot be compared with anything.
• Keep a frozen core (perhaps half the items) unchanged across refreshes, so there is always a like-for-like comparison across the boundary.
• Report both versions for one cycle. Old set 87%, new set 81%, both on the same system, and the six-point gap is a fact about the sets rather than a regression. Publish that gap; it is the most informative number in the whole refresh.
• Never swap silently. Every incident where a team argued for a week about a phantom regression traces back to somebody quietly adding forty items.

Retire items, without losing what they guard

An item that every version has passed for a year carries no information. It costs money and it never moves. But deleting it removes the guard against a regression that was fixed long ago, and fixed bugs do come back.

Move rather than delete. Retired items go to a cheap smoke suite that runs nightly with deterministic checks only, or in a cascade’s lowest tier. The regression is still guarded and the expensive judging budget goes to items that discriminate.

The reverse rule matters too: an item that flips constantly is either a rubric ambiguity or genuine non-determinism, and it belongs in quarantine rather than in the headline set.

Keep the composition honest

There is a specific way sets rot even when they are maintained. Every incident adds an item, so after two years the set is a museum of past bugs; and each of those items was chosen precisely because the system failed it, which makes the set systematically unrepresentative and systematically hard.

Hold a ratio. Something like: no more than a third of the set from incidents, the rest sampled from traffic on the refresh schedule. When incidents push past the ratio, either sample more traffic or move the older incident items into the regression suite, which is where they belonged all along.

The labels go stale too

A set can match today’s traffic perfectly and still be wrong, because the answers have moved. The refund window changed, a product was discontinued, a policy was rewritten, a fee went up. Every reference answer touching those topics is now incorrect, and the evaluation is quietly penalising the right behaviour.

Two defences. Attach a policy version to the items that depend on one, as the reference-answers lesson described, and fail the run loudly when the versions diverge. And when a regression appears on a cluster of related items at once (five refund questions failing on the same day) check the policy before you check the model. Roughly half the time the model is right and the file is old.

Give it an owner and a date

Sets rot when nobody owns them. Put a name and a review date in the set’s metadata file, review quarterly, and record what changed and why. It is fifteen minutes a quarter, and it is the difference between a measurement system and a folder of old JSON that everybody has stopped believing.