{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"human-oversight-that-is-real","title":"Measuring the human in the loop","url":"https://addaly.com/learn/evaluating-ai/human-oversight-that-is-real","position":64,"of":76,"minutes":9,"answer":"Oversight is a claim until you measure it, and the instrument is seeded errors, because reviewers catch fewer mistakes precisely as the system gets better and their attention drops.","keyIdea":"Oversight is a claim until you measure it, and the instrument is seeded errors, because reviewers catch fewer mistakes precisely as the system gets better and their attention drops."},"body":{"format":"markdown","text":"## The most common unmeasured claim in AI\n\n\"A human reviews every decision.\" It appears in system cards, in compliance documents, in sales decks, and it is almost never accompanied by a number. It is a statement about a workflow, not about an outcome, and the two come apart in a predictable direction.\n\n## Automation bias, and the paradox it creates\n\nPeople accept a confident machine suggestion more readily than they should, and more readily still when they are under time pressure, when the system is usually right, and when overriding takes effort. This direction of effect is well replicated across aviation, clinical decision support and driving automation, though the size varies a great deal by setting, so treat it as a reliable direction rather than a fixed number.\n\nThe consequence is a paradox worth stating plainly. **The better the system, the worse the reviewer.** At 99% accuracy, errors are rare, trust is high and attention drifts, so the rare error passes through. At 90% accuracy, the reviewer is engaged because they are correcting something every few minutes. A 99% model with rubber-stamp review can produce more errors reaching users than a 90% model with genuine review.\n\nThat means you cannot infer pipeline quality from model quality. You have to measure the pipeline.\n\n## Four numbers that describe the review\n\n- **Override rate.** What fraction of suggestions the reviewer changes. A rate near zero means either a very good system or no review at all, and you cannot tell which from this number alone.\n- **Override correctness.** Of the cases they changed, how often the change was an improvement. This distinguishes a careful reviewer from a nervous one, and it needs an independent adjudication of a sample.\n- **Miss rate.** Of the errors that were present, how many got through. This is the number that matters and it cannot be computed from ordinary traffic, because you do not know which items were wrong.\n- **Time per review**, plotted against accuracy. There is usually a cliff: below some number of seconds, catch rate collapses. Finding that cliff tells you the maximum sustainable queue rate, which is a staffing decision made with evidence.\n\n## Seeded errors are the instrument\n\nInject known-wrong suggestions into the review queue at a low, random rate (one or two per cent) and measure how many are caught. This is proficiency testing, used for decades in radiology quality assurance and in airport security screening, and it is the only way to get a real miss rate.\n\nHandle it properly, or you will destroy the trust the programme depends on:\n\n- Tell reviewers, in advance and in writing, that seeded cases exist and roughly how often.\n- Use the results to size and design the system, never to discipline an individual.\n- Seed realistic errors, not absurd ones, or you measure nothing.\n- Report the catch rate as a system property alongside model accuracy.\n\nA catch rate of 40% on seeded errors, with a 97% model, tells you the pipeline lets through more than you assumed. That single number has redirected more oversight designs than any amount of policy writing.\n\n## The conditions oversight actually needs\n\nReview that is real requires four things, and each is checkable:\n\n1. **Information.** The reviewer can see what the decision was based on: the source passage, the retrieved document, the fields extracted. Reviewing an answer without its evidence is guessing.\n2. **Authority.** Overriding is one action, not an escalation, and nothing in the interface treats it as an exception.\n3. **Time.** Queue pressure is measured, and the target rate sits on the good side of the accuracy cliff.\n4. **Incentives.** Nobody is measured on throughput alone. A reviewer paid per item will approve per item.\n\nAnd one design to avoid: **batch approval**. An interface where one click accepts forty items is not oversight, whatever the process document says. If batch approval is necessary for volume, sample from the batches and review those individually, and report the sampling rate as part of the oversight claim.\n\n## Report the pipeline, not the model\n\nThe honest external number is end-to-end: after the model and the reviewer together, what proportion of outputs reaching users are correct. It is usually higher than the model alone and lower than people expect, and it is the only figure that describes what a customer receives."},"check":{"question":"A team raises model accuracy from 92% to 98% and finds the end-to-end error rate reaching users barely improves. What is the most likely mechanism?","options":[{"text":"The reviewers were already catching nearly all errors, so there was little headroom for the model change to show.","correct":false,"tempting":"Would predict a low error rate before the change; the premise is that errors reaching users stayed roughly constant, which implies the reviewers were not catching most of them."},{"text":"The remaining 2% of errors are systematically harder cases that reviewers also find difficult.","correct":false,"tempting":"Plausible as a contributing factor and it does not explain why the six points of model improvement failed to translate at all."},{"text":"Errors are now rare enough that reviewer attention has dropped, so a larger share of the errors that remain pass through unchecked.","correct":true},{"text":"The evaluation set no longer represents production traffic, so the 98% figure is optimistic.","correct":false,"tempting":"A real hazard covered elsewhere in the course, and it would show as a gap between measured and observed model accuracy rather than in the oversight step."}]},"prev":{"slug":"evaluating-for-harm","title":"Measuring the worst case, not the average one","url":"https://addaly.com/learn/evaluating-ai/evaluating-for-harm"},"next":{"slug":"privacy-and-memorisation","title":"Testing whether the system gives away what it was given","url":"https://addaly.com/learn/evaluating-ai/privacy-and-memorisation"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Measuring the human in the loop', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/human-oversight-that-is-real (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Measuring the human in the loop","courseTitle":"Knowing If It Works","module":{"index":7,"slug":"assurance","title":"Evidence other people can act on"},"answer":null,"keyIdea":"Oversight is a claim until you measure it, and the instrument is seeded errors, because reviewers catch fewer mistakes precisely as the system gets better and their attention drops.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-most-common-unmeasured-claim-in-ai","runs":[{"text":"The most common unmeasured claim in AI"}]},{"t":"p","runs":[{"text":"“A human reviews every decision.” It appears in system cards, in compliance documents, in sales decks, and it is almost never accompanied by a number. It is a statement about a workflow, not about an outcome, and the two come apart in a predictable direction."}]},{"t":"h2","id":"automation-bias-and-the-paradox-it-creates","runs":[{"text":"Automation bias, and the paradox it creates"}]},{"t":"p","runs":[{"text":"People accept a confident machine suggestion more readily than they should, and more readily still when they are under time pressure, when the system is usually right, and when overriding takes effort. This direction of effect is well replicated across aviation, clinical decision support and driving automation, though the size varies a great deal by setting, so treat it as a reliable direction rather than a fixed number."}]},{"t":"p","runs":[{"text":"The consequence is a paradox worth stating plainly. "},{"text":"The better the system, the worse the reviewer.","b":true},{"text":" At 99% accuracy, errors are rare, trust is high and attention drifts, so the rare error passes through. At 90% accuracy, the reviewer is engaged because they are correcting something every few minutes. A 99% model with rubber-stamp review can produce more errors reaching users than a 90% model with genuine review."}]},{"t":"p","runs":[{"text":"That means you cannot infer pipeline quality from model quality. You have to measure the pipeline."}]},{"t":"h2","id":"four-numbers-that-describe-the-review","runs":[{"text":"Four numbers that describe the review"}]},{"t":"ul","items":[[{"text":"Override rate.","b":true},{"text":" What fraction of suggestions the reviewer changes. A rate near zero means either a very good system or no review at all, and you cannot tell which from this number alone."}],[{"text":"Override correctness.","b":true},{"text":" Of the cases they changed, how often the change was an improvement. This distinguishes a careful reviewer from a nervous one, and it needs an independent adjudication of a sample."}],[{"text":"Miss rate.","b":true},{"text":" Of the errors that were present, how many got through. This is the number that matters and it cannot be computed from ordinary traffic, because you do not know which items were wrong."}],[{"text":"Time per review","b":true},{"text":", plotted against accuracy. There is usually a cliff: below some number of seconds, catch rate collapses. Finding that cliff tells you the maximum sustainable queue rate, which is a staffing decision made with evidence."}]]},{"t":"h2","id":"seeded-errors-are-the-instrument","runs":[{"text":"Seeded errors are the instrument"}]},{"t":"p","runs":[{"text":"Inject known-wrong suggestions into the review queue at a low, random rate (one or two per cent) and measure how many are caught. This is proficiency testing, used for decades in radiology quality assurance and in airport security screening, and it is the only way to get a real miss rate."}]},{"t":"p","runs":[{"text":"Handle it properly, or you will destroy the trust the programme depends on:"}]},{"t":"ul","items":[[{"text":"Tell reviewers, in advance and in writing, that seeded cases exist and roughly how often."}],[{"text":"Use the results to size and design the system, never to discipline an individual."}],[{"text":"Seed realistic errors, not absurd ones, or you measure nothing."}],[{"text":"Report the catch rate as a system property alongside model accuracy."}]]},{"t":"p","runs":[{"text":"A catch rate of 40% on seeded errors, with a 97% model, tells you the pipeline lets through more than you assumed. That single number has redirected more oversight designs than any amount of policy writing."}]},{"t":"h2","id":"the-conditions-oversight-actually-needs","runs":[{"text":"The conditions oversight actually needs"}]},{"t":"p","runs":[{"text":"Review that is real requires four things, and each is checkable:"}]},{"t":"ol","start":1,"items":[[{"text":"Information.","b":true},{"text":" The reviewer can see what the decision was based on: the source passage, the retrieved document, the fields extracted. Reviewing an answer without its evidence is guessing."}],[{"text":"Authority.","b":true},{"text":" Overriding is one action, not an escalation, and nothing in the interface treats it as an exception."}],[{"text":"Time.","b":true},{"text":" Queue pressure is measured, and the target rate sits on the good side of the accuracy cliff."}],[{"text":"Incentives.","b":true},{"text":" Nobody is measured on throughput alone. A reviewer paid per item will approve per item."}]]},{"t":"p","runs":[{"text":"And one design to avoid: "},{"text":"batch approval","b":true},{"text":". An interface where one click accepts forty items is not oversight, whatever the process document says. If batch approval is necessary for volume, sample from the batches and review those individually, and report the sampling rate as part of the oversight claim."}]},{"t":"h2","id":"report-the-pipeline-not-the-model","runs":[{"text":"Report the pipeline, not the model"}]},{"t":"p","runs":[{"text":"The honest external number is end-to-end: after the model and the reviewer together, what proportion of outputs reaching users are correct. It is usually higher than the model alone and lower than people expect, and it is the only figure that describes what a customer receives."}]}],"check":{"question":[{"text":"A team raises model accuracy from 92% to 98% and finds the end-to-end error rate reaching users barely improves. What is the most likely mechanism?"}],"options":[{"text":[{"text":"The reviewers were already catching nearly all errors, so there was little headroom for the model change to show."}],"correct":false,"tempting":[{"text":"Would predict a low error rate before the change; the premise is that errors reaching users stayed roughly constant, which implies the reviewers were not catching most of them."}]},{"text":[{"text":"The remaining 2% of errors are systematically harder cases that reviewers also find difficult."}],"correct":false,"tempting":[{"text":"Plausible as a contributing factor and it does not explain why the six points of model improvement failed to translate at all."}]},{"text":[{"text":"Errors are now rare enough that reviewer attention has dropped, so a larger share of the errors that remain pass through unchecked."}],"correct":true,"tempting":null},{"text":[{"text":"The evaluation set no longer represents production traffic, so the 98% figure is optimistic."}],"correct":false,"tempting":[{"text":"A real hazard covered elsewhere in the course, and it would show as a gap between measured and observed model accuracy rather than in the oversight step."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"evaluating-for-harm","title":"Measuring the worst case, not the average one","minutes":9},"next":{"slug":"privacy-and-memorisation","title":"Testing whether the system gives away what it was given","minutes":9}}}