{"course":{"slug":"ai-and-you","title":"AI, Safety and What Goes Wrong","url":"https://addaly.com/learn/ai-and-you","level":"beginner","track":"start"},"lesson":{"slug":"automation-bias","title":"Why people believe the machine","url":"https://addaly.com/learn/ai-and-you/automation-bias","position":7,"of":73,"minutes":4,"answer":"A human check counts only with time, independent information, authority and a counted override rate.","keyIdea":"A human check counts only with time, independent information, authority and a counted override rate."},"body":{"format":"markdown","text":"## A computer against a village postmaster\n\nFrom 1999 the Post Office installed an accounting system called Horizon in its branches across the United Kingdom. It reported cash shortfalls that were not real. Sub-postmasters, often the most trusted person in a village, were told the computer showed money missing.\n\nBetween 2000 and 2014 the Post Office prosecuted about 736 sub-postmasters. Some went to prison. Some remortgaged their homes to repay money that had never gone anywhere. It took a civil trial brought by the sub-postmasters themselves, *Bates v Post Office*, for a court to find that Horizon had bugs, errors and defects capable of causing those shortfalls. The convictions were then quashed, first case by case in the Court of Appeal and then by an Act of Parliament.\n\n> Figure: Horizon: fifteen years of believing the computer. Timeline: 1999, Horizon installed in Post Office branches; 2000 to 2014, About 736 sub-postmasters prosecuted by the Post Office; Dec 2019, High Court: bugs in Horizon could cause the shortfalls; Apr 2021, Court of Appeal quashes 39 convictions; May 2024, An Act quashes convictions in England, Wales and Northern Ireland. Twenty years from installation to a court test of the software, and four and a half more before Parliament cleared the convictions by statute.\n\nFaced with a computer's figure and a person's account, an institution chose the computer, hundreds of times, over fifteen years. Two things made that easier. Evidence to the Post Office Horizon IT Inquiry showed that the Post Office and Fujitsu knew of bugs, and that branch accounts could be altered remotely, and did not tell the people they prosecuted. And the law in England and Wales presumes a computer was working properly unless someone shows otherwise, a presumption in place since Parliament repealed section 69 of the Police and Criminal Evidence Act in 1999. In 2025 the Ministry of Justice asked for evidence on changing it. A model that is wrong in fluent prose makes the institution's choice easier still.\n\n## Two kinds of error\n\nThe pattern has a name: **automation bias**. People accept a machine's recommendation more readily than a colleague's, and stop looking for evidence against it. A 2012 systematic review in the *Journal of the American Medical Informatics Association* found it across studies of clinical decision support.\n\nIt comes in two halves. **Commission errors** are doing what the system says when other information contradicts it. **Omission errors** are missing something because the system did not flag it. In a 2004 breast-screening study, readers using a computer's prompts caught fewer of the cancers it had failed to mark than readers working without it. A person told to check the output becomes a person who checks what the output points at.\n\nThe better the system usually is, the stronger the pull. A system right 99 times in 100 trains its reviewers to stop reviewing, just before the hundredth case.\n\n## Oversight or decoration\n\n\"A human reviews the output\" appears in almost every deployment plan. Whether it means anything depends on what the reviewer is given.\n\nStart with time. A reviewer with ninety seconds a case is a throughput device, so compare the time available with the time an independent view needs. Then information. A reviewer who sees only what the model saw, plus its answer, can disagree only on mood. Show them the source material, and let them form a view before the recommendation appears.\n\nAuthority comes next. If overriding needs a form and a senior signature while agreeing is free, you have built a rubber stamp. Last, count how often reviewers override. A rate of zero does not mean the model is perfect. It means review has stopped.\n\n## At your own desk\n\nForm your own answer before you look, even a rough one. It gives you something to notice a disagreement against.\n\nWhen you accept an output because checking is tedious, say so to yourself: I am accepting this because I do not want to check it. Sometimes the stakes do not justify a check. Then it is a decision rather than a drift."},"check":{"question":"A bank says its fraud reviewers overrode the model in 0 of 4,000 cases last quarter, and cites this as proof the model works well. What is the better reading?","options":[{"text":"The model is well calibrated, so it rarely needs human correction","correct":false,"tempting":"Reads zero overrides as model quality. It fits reviewers who have stopped forming a view just as well."},{"text":"The reviewers are well trained, so agreeing with a good model is expected","correct":false,"tempting":"Confuses agreement with checking. Reviewers who never disagree look the same as reviewers who never look."},{"text":"4,000 cases is too small a sample to establish any override rate","correct":false,"tempting":"Four thousand cases is plenty to see a real override rate. The striking fact is the exact zero."},{"text":"Zero overrides suggests review has stopped, and says little of the model","correct":true}],"why":"An independent reviewer disagrees sometimes. A rate of exactly zero means the check has become a stamp."},"prev":{"slug":"the-demo-and-the-machinery","title":"How a demo is built","url":"https://addaly.com/learn/ai-and-you/the-demo-and-the-machinery"},"next":{"slug":"the-liability-question","title":"Who is responsible when it is wrong","url":"https://addaly.com/learn/ai-and-you/the-liability-question"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Source and assessment review continues. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Why people believe the machine', in AI, Safety and What Goes Wrong. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-and-you/automation-bias (Accessed: 21 November 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Why people believe the machine","courseTitle":"AI, Safety and What Goes Wrong","module":{"index":1,"slug":"how-things-fail","title":"How these systems fail"},"answer":null,"keyIdea":"A human check counts only with time, independent information, authority and a counted override rate.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"caption":"Twenty years from installation to a court test of the software, and four and a half more before Parliament cleared the convictions by statute.","events":[{"when":"1999","what":"Horizon installed in Post Office branches"},{"when":"2000 to 2014","what":"About 736 sub-postmasters prosecuted by the Post Office"},{"when":"Dec 2019","what":"High Court: bugs in Horizon could cause the shortfalls"},{"when":"Apr 2021","what":"Court of Appeal quashes 39 convictions"},{"when":"May 2024","what":"An Act quashes convictions in England, Wales and Northern Ireland"}],"title":"Horizon: fifteen years of believing the computer","type":"timeline"},"alt":"Figure: Horizon: fifteen years of believing the computer. Timeline: 1999, Horizon installed in Post Office branches; 2000 to 2014, About 736 sub-postmasters prosecuted by the Post Office; Dec 2019, High Court: bugs in Horizon could cause the shortfalls; Apr 2021, Court of Appeal quashes 39 convictions; May 2024, An Act quashes convictions in England, Wales and Northern Ireland. Twenty years from installation to a court test of the software, and four and a half more before Parliament cleared the convictions by statute."}},"blocks":[{"t":"h2","id":"a-computer-against-a-village-postmaster","runs":[{"text":"A computer against a village postmaster"}]},{"t":"p","runs":[{"text":"From 1999 the Post Office installed an accounting system called Horizon in its branches across the United Kingdom. It reported cash shortfalls that were not real. Sub-postmasters, often the most trusted person in a village, were told the computer showed money missing."}]},{"t":"quick","prompt":"What kind of AI did Horizon use?","options":["A machine-learning model","An early expert system","None: it was accounting software"],"answer":2,"reveal":"None. Horizon was ordinary accounting software. What follows is what people did with its figures."},{"t":"p","runs":[{"text":"Between 2000 and 2014 the Post Office prosecuted about 736 sub-postmasters. Some went to prison. Some remortgaged their homes to repay money that had never gone anywhere. It took a civil trial brought by the sub-postmasters themselves, "},{"text":"Bates v Post Office","i":true},{"text":", for a court to find that Horizon had bugs, errors and defects capable of causing those shortfalls. The convictions were then quashed, first case by case in the Court of Appeal and then by an Act of Parliament."}]},{"t":"p","runs":[{"text":"Faced with a computer’s figure and a person’s account, an institution chose the computer, hundreds of times, over fifteen years. Two things made that easier. Evidence to the Post Office Horizon IT Inquiry showed that the Post Office and Fujitsu knew of bugs, and that branch accounts could be altered remotely, and did not tell the people they prosecuted. And the law in England and Wales presumes a computer was working properly unless someone shows otherwise, a presumption in place since Parliament repealed section 69 of the Police and Criminal Evidence Act in 1999. In 2025 the Ministry of Justice asked for evidence on changing it. A model that is wrong in fluent prose makes the institution’s choice easier still."}]},{"t":"h2","id":"two-kinds-of-error","runs":[{"text":"Two kinds of error"}]},{"t":"p","runs":[{"text":"The pattern has a name: "},{"text":"automation bias","b":true},{"text":". People accept a machine’s recommendation more readily than a colleague’s, and stop looking for evidence against it. A 2012 systematic review in the "},{"text":"Journal of the American Medical Informatics Association","i":true},{"text":" found it across studies of clinical decision support."}]},{"t":"p","runs":[{"text":"It comes in two halves. "},{"text":"Commission errors","b":true},{"text":" are doing what the system says when other information contradicts it. "},{"text":"Omission errors","b":true},{"text":" are missing something because the system did not flag it. In a 2004 breast-screening study, readers using a computer’s prompts caught fewer of the cancers it had failed to mark than readers working without it. A person told to check the output becomes a person who checks what the output points at."}]},{"t":"p","runs":[{"text":"The better the system usually is, the stronger the pull. A system right 99 times in 100 trains its reviewers to stop reviewing, just before the hundredth case."}]},{"t":"h2","id":"oversight-or-decoration","runs":[{"text":"Oversight or decoration"}]},{"t":"p","runs":[{"text":"“A human reviews the output” appears in almost every deployment plan. Whether it means anything depends on what the reviewer is given."}]},{"t":"p","runs":[{"text":"Start with time. A reviewer with ninety seconds a case is a throughput device, so compare the time available with the time an independent view needs. Then information. A reviewer who sees only what the model saw, plus its answer, can disagree only on mood. Show them the source material, and let them form a view before the recommendation appears."}]},{"t":"p","runs":[{"text":"Authority comes next. If overriding needs a form and a senior signature while agreeing is free, you have built a rubber stamp. Last, count how often reviewers override. A rate of zero does not mean the model is perfect. It means review has stopped."}]},{"t":"h2","id":"at-your-own-desk","runs":[{"text":"At your own desk"}]},{"t":"p","runs":[{"text":"Form your own answer before you look, even a rough one. It gives you something to notice a disagreement against."}]},{"t":"p","runs":[{"text":"When you accept an output because checking is tedious, say so to yourself: I am accepting this because I do not want to check it. Sometimes the stakes do not justify a check. Then it is a decision rather than a drift."}]}],"check":{"question":[{"text":"A bank says its fraud reviewers overrode the model in 0 of 4,000 cases last quarter, and cites this as proof the model works well. What is the better reading?"}],"options":[{"text":[{"text":"The model is well calibrated, so it rarely needs human correction"}],"correct":false,"tempting":[{"text":"Reads zero overrides as model quality. It fits reviewers who have stopped forming a view just as well."}]},{"text":[{"text":"The reviewers are well trained, so agreeing with a good model is expected"}],"correct":false,"tempting":[{"text":"Confuses agreement with checking. Reviewers who never disagree look the same as reviewers who never look."}]},{"text":[{"text":"4,000 cases is too small a sample to establish any override rate"}],"correct":false,"tempting":[{"text":"Four thousand cases is plenty to see a real override rate. The striking fact is the exact zero."}]},{"text":[{"text":"Zero overrides suggests review has stopped, and says little of the model"}],"correct":true,"tempting":null}],"order":[2,1,0,3],"why":[{"text":"An independent reviewer disagrees sometimes. A rate of exactly zero means the check has become a stamp."}]},"prev":{"slug":"the-demo-and-the-machinery","title":"How a demo is built","minutes":3},"next":{"slug":"the-liability-question","title":"Who is responsible when it is wrong","minutes":4}}}