A computer against a village postmaster
From 1999 the Post Office installed an accounting system called Horizon in its branches across the United Kingdom. It reported cash shortfalls that were not real. Sub-postmasters, often the most trusted person in a village, were told the computer showed money missing.
Quick check
What kind of AI did Horizon use?
Between 2000 and 2014 the Post Office prosecuted about 736 sub-postmasters. Some went to prison. Some remortgaged their homes to repay money that had never gone anywhere. It took a civil trial brought by the sub-postmasters themselves, Bates v Post Office, for a court to find that Horizon had bugs, errors and defects capable of causing those shortfalls. The convictions were then quashed, first case by case in the Court of Appeal and then by an Act of Parliament.
Faced with a computer’s figure and a person’s account, an institution chose the computer, hundreds of times, over fifteen years. Two things made that easier. Evidence to the Post Office Horizon IT Inquiry showed that the Post Office and Fujitsu knew of bugs, and that branch accounts could be altered remotely, and did not tell the people they prosecuted. And the law in England and Wales presumes a computer was working properly unless someone shows otherwise, a presumption in place since Parliament repealed section 69 of the Police and Criminal Evidence Act in 1999. In 2025 the Ministry of Justice asked for evidence on changing it. A model that is wrong in fluent prose makes the institution’s choice easier still.
Two kinds of error
The pattern has a name: automation bias. People accept a machine’s recommendation more readily than a colleague’s, and stop looking for evidence against it. A 2012 systematic review in the Journal of the American Medical Informatics Association found it across studies of clinical decision support.
It comes in two halves. Commission errors are doing what the system says when other information contradicts it. Omission errors are missing something because the system did not flag it. In a 2004 breast-screening study, readers using a computer’s prompts caught fewer of the cancers it had failed to mark than readers working without it. A person told to check the output becomes a person who checks what the output points at.
The better the system usually is, the stronger the pull. A system right 99 times in 100 trains its reviewers to stop reviewing, just before the hundredth case.
Oversight or decoration
“A human reviews the output” appears in almost every deployment plan. Whether it means anything depends on what the reviewer is given.
Start with time. A reviewer with ninety seconds a case is a throughput device, so compare the time available with the time an independent view needs. Then information. A reviewer who sees only what the model saw, plus its answer, can disagree only on mood. Show them the source material, and let them form a view before the recommendation appears.
Authority comes next. If overriding needs a form and a senior signature while agreeing is free, you have built a rubber stamp. Last, count how often reviewers override. A rate of zero does not mean the model is perfect. It means review has stopped.
At your own desk
Form your own answer before you look, even a rough one. It gives you something to notice a disagreement against.
When you accept an output because checking is tedious, say so to yourself: I am accepting this because I do not want to check it. Sometimes the stakes do not justify a check. Then it is a decision rather than a drift.
Check your understanding
A bank says its fraud reviewers overrode the model in 0 of 4,000 cases last quarter, and cites this as proof the model works well. What is the better reading?
Pick the one you would defend. Your answer stays private.
Your place and first choices stay on this browser. Classroom credit is recorded separately.
Stuck on something?Ask Rukmini, your AI tutor