{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"human-review-that-scales","title":"Sampling live traffic, and keeping reviewers honest","url":"https://addaly.com/learn/evaluating-ai/human-review-that-scales","position":58,"of":76,"minutes":9,"answer":"Review a random sample as well as a flagged one, because the flagged sample only ever teaches you about failures you can already detect.","keyIdea":"Review a random sample as well as a flagged one, because the flagged sample only ever teaches you about failures you can already detect."},"body":{"format":"markdown","text":"## Automation has a floor\n\nDeterministic checks catch the mechanical failures. A calibrated judge catches the criteria you have already written down. Neither catches the failure nobody has thought of yet, and that is the one that ends up on social media.\n\nSo some human reading of live traffic is permanent. The engineering question is how to get the most out of a small, fixed amount of it: realistically one person for an hour a day, not a review team.\n\n## Two samples, never one\n\n**A random sample.** Twenty items a day, drawn uniformly from all traffic, no filtering. This is the only thing that gives you an unbiased error rate and the only thing that finds unknown failure modes. It is also boring, because most of it is fine, and it is therefore the first thing cut. Protect it.\n\n**A targeted sample.** Twenty more, drawn from where the problems are: items the judge flagged, items with low retrieval scores, items that were regenerated, sessions that escalated, anything a user complained about. This is where you find the most bugs per minute.\n\nReview only the targeted sample and you learn about the failures you can already detect, which is a closed loop that congratulates itself. The random sample is what breaks it open. A useful diagnostic: count how many *new* failure kinds each sample produced this month. When the random sample stops producing new kinds, your automated checks have caught up, and that is worth knowing.\n\n## Forty items a day is a real programme\n\nDo the arithmetic before designing anything elaborate. Two minutes per item, forty items, is eighty minutes a day. Over a month that is roughly 900 reviewed items, which at typical volumes is a one to two per cent sample and enough to estimate an error rate to within about three points.\n\nThat is a modest, sustainable commitment that most teams can actually keep, and it beats an ambitious review process that is abandoned in week three.\n\n## Keep the reviewers calibrated too\n\nReviewers drift. They get faster, they get more forgiving, they develop private conventions. Three cheap controls:\n\n- **Overlap 10%.** One item in ten goes to two reviewers. Track agreement over time. A falling kappa is drift, and it shows up weeks before anyone would have noticed by feel.\n- **Salt the queue.** Every so often, insert an item that was labelled months ago and whose answer is known. If today's verdict differs from the archived one, the standard has moved, not the traffic.\n- **Run a short calibration meeting.** Half an hour a fortnight, five disagreements, and a written ruling added to the guide for each. This is the only mechanism that keeps the guide alive.\n\n## Design the queue so it is quick to be right\n\nThe interface decides the throughput more than the reviewer's speed does. The rules that matter:\n\n- **One item, one screen.** The input, the output, the retrieved context, and the buttons. No scrolling to reach the verdict.\n- **Ask the rubric's questions, not \"is this good\".** Four checkboxes matching your rules, not a five-star widget. Same reason as in module three: a fused score cannot be acted on.\n- **A free-text box for \"something else was wrong\".** This field is where new failure modes come from. Read it weekly, and treat a recurring phrase in it as a candidate rubric rule.\n- **Show the item's provenance** (random or targeted, and why it was flagged) but *after* the verdict, not before, or you have primed the reviewer.\n\n**Label Studio** and **Argilla** are both free and open source and do all of this out of the box, including blind double-review and agreement reports. A spreadsheet with a form front-end works at forty items a day and costs nothing at all.\n\n## The output is not a score\n\nThe point of the review programme is not the weekly quality percentage. It is three things:\n\n1. **New eval items**, with expected outputs, added to the set.\n2. **New rubric rules**, from failures your criteria did not cover.\n3. **A calibration signal for the judge**: reviewed items are exactly the labelled data your judge should be re-measured against, at no extra cost.\n\nThat third point closes the loop cleanly. Live review generates labels; labels calibrate the judge; the judge scales to all traffic; its flags target tomorrow's review. Each layer feeds the next, and the human hour a day is what keeps the whole thing anchored to reality rather than to its own past.\n\nOne caution to end on. Reviewing your own product's failures every day is dispiriting if it is framed as quality control on people. Frame it as the team's route to the next fix, rotate it, and share what each week's review changed. A review programme that nobody wants to do lasts about a month."},"check":{"question":"A team reviews 50 production items a week, all drawn from cases their judge flagged as low quality. After three months their error rate looks stable and no new failure types have appeared. What is the most likely reason?","options":[{"text":"The system has genuinely stabilised, and the review budget could now be reduced.","correct":false,"tempting":"Reads an absence of new findings as evidence of quality when the sampling method could not have produced new findings either way."},{"text":"The sample only contains failures the judge already recognises, so it cannot surface anything the judge misses.","correct":true},{"text":"Fifty items a week is too small a sample to detect new failure types at typical rates.","correct":false,"tempting":"Blames sample size when the same 50 items drawn at random would have been capable of surfacing unknown failures."},{"text":"Reviewer drift has made the reviewers more lenient over three months, hiding a real decline.","correct":false,"tempting":"Names a genuine hazard that overlap checks exist to catch, but it would not explain the absence of new failure categories."}]},"prev":{"slug":"online-metrics-and-proxies","title":"The signals users give you without being asked","url":"https://addaly.com/learn/evaluating-ai/online-metrics-and-proxies"},"next":{"slug":"writing-a-system-card","title":"The document that travels with the system","url":"https://addaly.com/learn/evaluating-ai/writing-a-system-card"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Sampling live traffic, and keeping reviewers honest', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/human-review-that-scales (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Sampling live traffic, and keeping reviewers honest","courseTitle":"Knowing If It Works","module":{"index":6,"slug":"evaluation-in-the-loop","title":"Evaluation in the working loop"},"answer":null,"keyIdea":"Review a random sample as well as a flagged one, because the flagged sample only ever teaches you about failures you can already detect.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"automation-has-a-floor","runs":[{"text":"Automation has a floor"}]},{"t":"p","runs":[{"text":"Deterministic checks catch the mechanical failures. A calibrated judge catches the criteria you have already written down. Neither catches the failure nobody has thought of yet, and that is the one that ends up on social media."}]},{"t":"p","runs":[{"text":"So some human reading of live traffic is permanent. The engineering question is how to get the most out of a small, fixed amount of it: realistically one person for an hour a day, not a review team."}]},{"t":"h2","id":"two-samples-never-one","runs":[{"text":"Two samples, never one"}]},{"t":"p","runs":[{"text":"A random sample.","b":true},{"text":" Twenty items a day, drawn uniformly from all traffic, no filtering. This is the only thing that gives you an unbiased error rate and the only thing that finds unknown failure modes. It is also boring, because most of it is fine, and it is therefore the first thing cut. Protect it."}]},{"t":"p","runs":[{"text":"A targeted sample.","b":true},{"text":" Twenty more, drawn from where the problems are: items the judge flagged, items with low retrieval scores, items that were regenerated, sessions that escalated, anything a user complained about. This is where you find the most bugs per minute."}]},{"t":"p","runs":[{"text":"Review only the targeted sample and you learn about the failures you can already detect, which is a closed loop that congratulates itself. The random sample is what breaks it open. A useful diagnostic: count how many "},{"text":"new","i":true},{"text":" failure kinds each sample produced this month. When the random sample stops producing new kinds, your automated checks have caught up, and that is worth knowing."}]},{"t":"h2","id":"forty-items-a-day-is-a-real-programme","runs":[{"text":"Forty items a day is a real programme"}]},{"t":"p","runs":[{"text":"Do the arithmetic before designing anything elaborate. Two minutes per item, forty items, is eighty minutes a day. Over a month that is roughly 900 reviewed items, which at typical volumes is a one to two per cent sample and enough to estimate an error rate to within about three points."}]},{"t":"p","runs":[{"text":"That is a modest, sustainable commitment that most teams can actually keep, and it beats an ambitious review process that is abandoned in week three."}]},{"t":"h2","id":"keep-the-reviewers-calibrated-too","runs":[{"text":"Keep the reviewers calibrated too"}]},{"t":"p","runs":[{"text":"Reviewers drift. They get faster, they get more forgiving, they develop private conventions. Three cheap controls:"}]},{"t":"ul","items":[[{"text":"Overlap 10%.","b":true},{"text":" One item in ten goes to two reviewers. Track agreement over time. A falling kappa is drift, and it shows up weeks before anyone would have noticed by feel."}],[{"text":"Salt the queue.","b":true},{"text":" Every so often, insert an item that was labelled months ago and whose answer is known. If today’s verdict differs from the archived one, the standard has moved, not the traffic."}],[{"text":"Run a short calibration meeting.","b":true},{"text":" Half an hour a fortnight, five disagreements, and a written ruling added to the guide for each. This is the only mechanism that keeps the guide alive."}]]},{"t":"h2","id":"design-the-queue-so-it-is-quick-to-be-right","runs":[{"text":"Design the queue so it is quick to be right"}]},{"t":"p","runs":[{"text":"The interface decides the throughput more than the reviewer’s speed does. The rules that matter:"}]},{"t":"ul","items":[[{"text":"One item, one screen.","b":true},{"text":" The input, the output, the retrieved context, and the buttons. No scrolling to reach the verdict."}],[{"text":"Ask the rubric’s questions, not “is this good”.","b":true},{"text":" Four checkboxes matching your rules, not a five-star widget. Same reason as in module three: a fused score cannot be acted on."}],[{"text":"A free-text box for “something else was wrong”.","b":true},{"text":" This field is where new failure modes come from. Read it weekly, and treat a recurring phrase in it as a candidate rubric rule."}],[{"text":"Show the item’s provenance","b":true},{"text":" (random or targeted, and why it was flagged) but "},{"text":"after","i":true},{"text":" the verdict, not before, or you have primed the reviewer."}]]},{"t":"p","runs":[{"text":"Label Studio","b":true},{"text":" and "},{"text":"Argilla","b":true},{"text":" are both free and open source and do all of this out of the box, including blind double-review and agreement reports. A spreadsheet with a form front-end works at forty items a day and costs nothing at all."}]},{"t":"h2","id":"the-output-is-not-a-score","runs":[{"text":"The output is not a score"}]},{"t":"p","runs":[{"text":"The point of the review programme is not the weekly quality percentage. It is three things:"}]},{"t":"ol","start":1,"items":[[{"text":"New eval items","b":true},{"text":", with expected outputs, added to the set."}],[{"text":"New rubric rules","b":true},{"text":", from failures your criteria did not cover."}],[{"text":"A calibration signal for the judge","b":true},{"text":": reviewed items are exactly the labelled data your judge should be re-measured against, at no extra cost."}]]},{"t":"p","runs":[{"text":"That third point closes the loop cleanly. Live review generates labels; labels calibrate the judge; the judge scales to all traffic; its flags target tomorrow’s review. Each layer feeds the next, and the human hour a day is what keeps the whole thing anchored to reality rather than to its own past."}]},{"t":"p","runs":[{"text":"One caution to end on. Reviewing your own product’s failures every day is dispiriting if it is framed as quality control on people. Frame it as the team’s route to the next fix, rotate it, and share what each week’s review changed. A review programme that nobody wants to do lasts about a month."}]}],"check":{"question":[{"text":"A team reviews 50 production items a week, all drawn from cases their judge flagged as low quality. After three months their error rate looks stable and no new failure types have appeared. What is the most likely reason?"}],"options":[{"text":[{"text":"The system has genuinely stabilised, and the review budget could now be reduced."}],"correct":false,"tempting":[{"text":"Reads an absence of new findings as evidence of quality when the sampling method could not have produced new findings either way."}]},{"text":[{"text":"The sample only contains failures the judge already recognises, so it cannot surface anything the judge misses."}],"correct":true,"tempting":null},{"text":[{"text":"Fifty items a week is too small a sample to detect new failure types at typical rates."}],"correct":false,"tempting":[{"text":"Blames sample size when the same 50 items drawn at random would have been capable of surfacing unknown failures."}]},{"text":[{"text":"Reviewer drift has made the reviewers more lenient over three months, hiding a real decline."}],"correct":false,"tempting":[{"text":"Names a genuine hazard that overlap checks exist to catch, but it would not explain the absence of new failure categories."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"online-metrics-and-proxies","title":"The signals users give you without being asked","minutes":9},"next":{"slug":"writing-a-system-card","title":"The document that travels with the system","minutes":9}}}