Why people believe the machine

The safeguard that is not one

Almost every plan for deploying AI safely contains the phrase "a human reviews the output". It is comforting, cheap to write, and — as designed in most organisations — close to worthless. The reason is a well-studied phenomenon called automation bias: people accept a machine's recommendation more readily than they would accept the same recommendation from a colleague, and they stop looking for evidence against it.

It has two halves. Commission errors: doing what the system says even when other information contradicts it. Omission errors: missing something because the system did not flag it. The second is the quieter and more common failure. A human told to check a system's output becomes a human who checks whatever the system pointed at.

The research base goes back to aviation and clinical decision support. Studies of computer-aided detection in mammography found that readers' behaviour reorganised around the prompts: sensitivity rose for marked regions and fell for unmarked ones. In simulated flight decks, crews have followed erroneous automated guidance in the presence of contradicting instruments. The pattern is robust, it is not a sign of stupidity, and it gets stronger the more reliable the system usually is — which is the cruel part. A system that is right 99% of the time trains its reviewers to stop reviewing, precisely so that they are unprepared for the 1%.

The clearest case is not about AI

The British Post Office scandal is the most complete demonstration available, and it involves no machine learning at all.

From 1999 the Post Office rolled out an accounting system called Horizon across its branches. It produced shortfalls that were not real. Sub-postmasters — people running small local branches, often for decades, often the most trusted person in a village — were told the computer showed money missing. Over 900 were prosecuted. People were imprisoned. Some remortgaged houses to repay money that had never gone anywhere. At least one person took their own life. Convictions were finally quashed by legislation in 2024 and a public inquiry has examined how it happened.

The mechanism at the centre is the one this lesson is about. Faced with a discrepancy between what a computer reported and what a person said, an institution repeatedly chose the computer. Not once, in a moment of confusion — hundreds of times, over fifteen years, against hundreds of individually plausible people whose accounts were consistent with each other. The system's output was treated as evidence and the human's account was treated as a claim.

No machine learning was needed to produce that. Adding a model that is confidently wrong in fluent prose does not improve the situation.

What real oversight requires

If you are designing a process, four conditions separate oversight from decoration.

Time. A reviewer with ninety seconds per case is a throughput device. Measure the time actually available per item and compare it against the time genuinely needed to reach an independent view.

Independent information. If the reviewer sees only what the model saw, plus the model's answer, they cannot disagree on any basis except mood. Give them the source material and, ideally, let them form a view before revealing the recommendation. Ordering matters enormously: the recommendation shown first becomes the anchor.

Authority. Can the reviewer override without justifying themselves to someone senior? If overriding is administratively expensive and agreeing is free, the system's recommendation is the default and you have built a rubber stamp with a job title.

Measurement. Track the override rate. An override rate of zero does not mean the model is perfect; it means the review is not happening. This is the single most useful number in any human-in-the-loop deployment, and almost nobody collects it.

For you personally

The same bias operates on one person at a desk. Two habits help, and both cost seconds.

Form your own answer before you look. Even a rough one. It gives you something to notice a disagreement against, which is exactly what the anchoring effect removes.

And when you find yourself accepting an output because checking it is tedious, say the reason out loud: I am accepting this because I do not want to check it. That sentence is uncomfortable enough to be useful. Sometimes the honest answer is that the stakes do not justify checking, and that is fine — it is a decision rather than a drift.