Measuring the human in the loop

The most common unmeasured claim in AI

“A human reviews every decision.” It appears in system cards, in compliance documents, in sales decks, and it is almost never accompanied by a number. It is a statement about a workflow, not about an outcome, and the two come apart in a predictable direction.

Automation bias, and the paradox it creates

People accept a confident machine suggestion more readily than they should, and more readily still when they are under time pressure, when the system is usually right, and when overriding takes effort. This direction of effect is well replicated across aviation, clinical decision support and driving automation, though the size varies a great deal by setting, so treat it as a reliable direction rather than a fixed number.

The consequence is a paradox worth stating plainly. The better the system, the worse the reviewer. At 99% accuracy, errors are rare, trust is high and attention drifts, so the rare error passes through. At 90% accuracy, the reviewer is engaged because they are correcting something every few minutes. A 99% model with rubber-stamp review can produce more errors reaching users than a 90% model with genuine review.

That means you cannot infer pipeline quality from model quality. You have to measure the pipeline.

Four numbers that describe the review

• Override rate. What fraction of suggestions the reviewer changes. A rate near zero means either a very good system or no review at all, and you cannot tell which from this number alone.
• Override correctness. Of the cases they changed, how often the change was an improvement. This distinguishes a careful reviewer from a nervous one, and it needs an independent adjudication of a sample.
• Miss rate. Of the errors that were present, how many got through. This is the number that matters and it cannot be computed from ordinary traffic, because you do not know which items were wrong.
• Time per review, plotted against accuracy. There is usually a cliff: below some number of seconds, catch rate collapses. Finding that cliff tells you the maximum sustainable queue rate, which is a staffing decision made with evidence.

Seeded errors are the instrument

Inject known-wrong suggestions into the review queue at a low, random rate (one or two per cent) and measure how many are caught. This is proficiency testing, used for decades in radiology quality assurance and in airport security screening, and it is the only way to get a real miss rate.

Handle it properly, or you will destroy the trust the programme depends on:

• Tell reviewers, in advance and in writing, that seeded cases exist and roughly how often.
• Use the results to size and design the system, never to discipline an individual.
• Seed realistic errors, not absurd ones, or you measure nothing.
• Report the catch rate as a system property alongside model accuracy.

A catch rate of 40% on seeded errors, with a 97% model, tells you the pipeline lets through more than you assumed. That single number has redirected more oversight designs than any amount of policy writing.

The conditions oversight actually needs

Review that is real requires four things, and each is checkable:

1. Information. The reviewer can see what the decision was based on: the source passage, the retrieved document, the fields extracted. Reviewing an answer without its evidence is guessing.
2. Authority. Overriding is one action, not an escalation, and nothing in the interface treats it as an exception.
3. Time. Queue pressure is measured, and the target rate sits on the good side of the accuracy cliff.
4. Incentives. Nobody is measured on throughput alone. A reviewer paid per item will approve per item.

And one design to avoid: batch approval. An interface where one click accepts forty items is not oversight, whatever the process document says. If batch approval is necessary for volume, sample from the batches and review those individually, and report the sampling rate as part of the oversight claim.

Report the pipeline, not the model

The honest external number is end-to-end: after the model and the reviewer together, what proportion of outputs reaching users are correct. It is usually higher than the model alone and lower than people expect, and it is the only figure that describes what a customer receives.