Sampling live traffic, and keeping reviewers honest

Automation has a floor

Deterministic checks catch the mechanical failures. A calibrated judge catches the criteria you have already written down. Neither catches the failure nobody has thought of yet, and that is the one that ends up on social media.

So some human reading of live traffic is permanent. The engineering question is how to get the most out of a small, fixed amount of it: realistically one person for an hour a day, not a review team.

Two samples, never one

A random sample. Twenty items a day, drawn uniformly from all traffic, no filtering. This is the only thing that gives you an unbiased error rate and the only thing that finds unknown failure modes. It is also boring, because most of it is fine, and it is therefore the first thing cut. Protect it.

A targeted sample. Twenty more, drawn from where the problems are: items the judge flagged, items with low retrieval scores, items that were regenerated, sessions that escalated, anything a user complained about. This is where you find the most bugs per minute.

Review only the targeted sample and you learn about the failures you can already detect, which is a closed loop that congratulates itself. The random sample is what breaks it open. A useful diagnostic: count how many new failure kinds each sample produced this month. When the random sample stops producing new kinds, your automated checks have caught up, and that is worth knowing.

Forty items a day is a real programme

Do the arithmetic before designing anything elaborate. Two minutes per item, forty items, is eighty minutes a day. Over a month that is roughly 900 reviewed items, which at typical volumes is a one to two per cent sample and enough to estimate an error rate to within about three points.

That is a modest, sustainable commitment that most teams can actually keep, and it beats an ambitious review process that is abandoned in week three.

Keep the reviewers calibrated too

Reviewers drift. They get faster, they get more forgiving, they develop private conventions. Three cheap controls:

• Overlap 10%. One item in ten goes to two reviewers. Track agreement over time. A falling kappa is drift, and it shows up weeks before anyone would have noticed by feel.
• Salt the queue. Every so often, insert an item that was labelled months ago and whose answer is known. If today’s verdict differs from the archived one, the standard has moved, not the traffic.
• Run a short calibration meeting. Half an hour a fortnight, five disagreements, and a written ruling added to the guide for each. This is the only mechanism that keeps the guide alive.

Design the queue so it is quick to be right

The interface decides the throughput more than the reviewer’s speed does. The rules that matter:

• One item, one screen. The input, the output, the retrieved context, and the buttons. No scrolling to reach the verdict.
• Ask the rubric’s questions, not “is this good”. Four checkboxes matching your rules, not a five-star widget. Same reason as in module three: a fused score cannot be acted on.
• A free-text box for “something else was wrong”. This field is where new failure modes come from. Read it weekly, and treat a recurring phrase in it as a candidate rubric rule.
• Show the item’s provenance (random or targeted, and why it was flagged) but after the verdict, not before, or you have primed the reviewer.

Label Studio and Argilla are both free and open source and do all of this out of the box, including blind double-review and agreement reports. A spreadsheet with a form front-end works at forty items a day and costs nothing at all.

The output is not a score

The point of the review programme is not the weekly quality percentage. It is three things:

1. New eval items, with expected outputs, added to the set.
2. New rubric rules, from failures your criteria did not cover.
3. A calibration signal for the judge: reviewed items are exactly the labelled data your judge should be re-measured against, at no extra cost.

That third point closes the loop cleanly. Live review generates labels; labels calibrate the judge; the judge scales to all traffic; its flags target tomorrow’s review. Each layer feeds the next, and the human hour a day is what keeps the whole thing anchored to reality rather than to its own past.

One caution to end on. Reviewing your own product’s failures every day is dispiriting if it is framed as quality control on people. Frame it as the team’s route to the next fix, rotate it, and share what each week’s review changed. A review programme that nobody wants to do lasts about a month.