{"course":{"slug":"ai-and-you","title":"AI, Safety and What Goes Wrong","url":"https://addaly.com/learn/ai-and-you","level":"beginner","track":"start"},"lesson":{"slug":"what-actually-goes-wrong","title":"Design, capability or misuse","url":"https://addaly.com/learn/ai-and-you/what-actually-goes-wrong","position":1,"of":73,"minutes":3,"answer":"Every failure is design, capability or misuse, and each needs a different fix.","keyIdea":"Every failure is design, capability or misuse, and each needs a different fix."},"body":{"format":"markdown","text":"In March 2023 two New York lawyers filed a brief in *Mata v. Avianca*, a personal injury claim against an airline in the federal court in Manhattan. Six of the decisions it cited did not exist. ChatGPT had written them, with judges' names, docket numbers and quotations. When one lawyer asked it whether a case was real, it said yes.\n\nThat is one failure. The news carries dozens a week, and they are not all the same kind. Before reading on, sort six real headlines into three bins.\n\n## Three bins, three fixes\n\n**Design.** The system works as built, and the build is the problem. Amazon's experimental hiring tool learned from ten years of CVs, most of them from men, and marked down the word \"women's\". It learned what it was shown. An illustrative case makes the same point and runs through this course's film: a bank's loan model turns down a whole district, because few loans were ever made there, so few repayments were ever recorded. The fix is a different target, different data or a different threshold.\n\n**Capability.** The system fails at the thing it claims to do. ChatGPT is not a legal database. It produces plausible text, and a plausible citation is what Steven Schwartz asked it for. The fix sits outside the tool: read every citation in a primary source before you sign.\n\n> Figure: Asked if its case was real, and what the court found. What ChatGPT told Steven Schwartz: Varghese \"does indeed exist\"; It is on Westlaw and LexisNexis; The other cases are \"real\" authorities too. What the court found, 22 June 2023: The Eleventh Circuit's clerk: no such ruling; Its legal analysis is \"gibberish\"; Six cited decisions did not exist; A $5,000 penalty on the lawyers and firm. From Judge Castel's opinion and order on sanctions. The reply about the citation came from the process that wrote the citation, so it checked nothing.\n\n**Misuse.** The system works, and somebody points it at you. In 2024 a finance worker in Hong Kong paid out about $25 million after a video call in which every colleague was a deepfake. The fix is procedure: hang up and call back on a number you already hold.\n\n## Find the person who chose\n\nWhen a system turns you down, no machine decided. People chose the model, the data, the target, the score at which a yes becomes a no, and whether anyone looks again. \"The algorithm decided\" moves those choices from people to software.\n\nIn *Mata* the people are easy to name. One lawyer used a tool he had never used before. Another signed the brief without reading the cases. When the airline said the cases could not be found, both stood by them. On 22 June 2023 Judge P. Kevin Castel fined them and their firm $5,000. He wrote that using a reliable AI tool is not improper in itself, and that the rules make lawyers the gatekeepers of what they file.\n\nSo ask two questions of any failure. Which bin is it in? Which human decision put it there? The whole Mata story, with the choice the firm faced in April 2023, is this module's case study."},"check":{"question":"A delivery firm's routing model sends vans past a primary school at pick-up time, because it was told to minimise minutes and nothing else. Which kind of failure is this?","options":[{"text":"Capability: the model failed to find the best route for vans","correct":false,"tempting":"The route looks like a mistake, so it feels as if the tool fell short. It found exactly the fastest route it was asked for."},{"text":"Misuse: somebody aimed the routing tool at the schoolchildren","correct":false,"tempting":"The harm lands on children who never chose the tool, which feels like being targeted. Nobody pointed it at anyone."},{"text":"Design: it met its target, and the target left out safety","correct":true},{"text":"Capability: the map data must have been old or incomplete","correct":false,"tempting":"Old maps do cause bad routes. Nothing here says the map was wrong, and the route really was the fastest."}],"why":"Minutes were the only thing it was told to value. The fix is a different target, and a person chooses it."},"prev":null,"next":{"slug":"what-the-machine-is-doing","title":"What the model computes","url":"https://addaly.com/learn/ai-and-you/what-the-machine-is-doing"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Design, capability or misuse', in AI, Safety and What Goes Wrong. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-and-you/what-actually-goes-wrong (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Design, capability or misuse","courseTitle":"AI, Safety and What Goes Wrong","module":{"index":1,"slug":"how-things-fail","title":"How these systems fail"},"answer":null,"keyIdea":"Every failure is design, capability or misuse, and each needs a different fix.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 1.1","figure":{"left":{"title":"What ChatGPT told Steven Schwartz","items":["Varghese “does indeed exist”","It is on Westlaw and LexisNexis","The other cases are “real” authorities too"]},"type":"compare","right":{"title":"What the court found, 22 June 2023","items":["The Eleventh Circuit’s clerk: no such ruling","Its legal analysis is “gibberish”","Six cited decisions did not exist","A $5,000 penalty on the lawyers and firm"]},"title":"Asked if its case was real, and what the court found","caption":"From Judge Castel’s opinion and order on sanctions. The reply about the citation came from the process that wrote the citation, so it checked nothing."},"alt":"Figure: Asked if its case was real, and what the court found. What ChatGPT told Steven Schwartz: Varghese “does indeed exist”; It is on Westlaw and LexisNexis; The other cases are “real” authorities too. What the court found, 22 June 2023: The Eleventh Circuit’s clerk: no such ruling; Its legal analysis is “gibberish”; Six cited decisions did not exist; A $5,000 penalty on the lawyers and firm. From Judge Castel’s opinion and order on sanctions. The reply about the citation came from the process that wrote the citation, so it checked nothing."}},"blocks":[{"t":"p","runs":[{"text":"In March 2023 two New York lawyers filed a brief in "},{"text":"Mata v. Avianca","i":true},{"text":", a personal injury claim against an airline in the federal court in Manhattan. Six of the decisions it cited did not exist. ChatGPT had written them, with judges’ names, docket numbers and quotations. When one lawyer asked it whether a case was real, it said yes."}],"lede":true},{"t":"p","runs":[{"text":"That is one failure. The news carries dozens a week, and they are not all the same kind. Before reading on, sort six real headlines into three bins."}]},{"t":"sort","prompt":"Six real headlines. Put each one in the bin where its failure comes from.","buckets":["Design","Capability","Misuse"],"legend":[{"bucket":"Design","meaning":"it worked as built, and the design is the problem."},{"bucket":"Capability","meaning":"it failed at the job it claims to do."},{"bucket":"Misuse","meaning":"it worked, and someone aimed it at people."}],"items":[{"text":"Amazon scraps secret AI recruiting tool that showed bias against women","answer":"Design","note":"The model worked as built. The design is the problem: what it learned from, or what it was told to aim for.","source":{"title":"Reuters, 10 October 2018","url":"https://www.reuters.com/article/us-amazon-com-jobs-automation-insight-idUSKCN1MK08G"}},{"text":"NYC’s AI chatbot tells businesses to break the law","answer":"Capability","note":"The system failed at the job it claims to do. It gave a plausible answer that was not true.","source":{"title":"The Markup, 29 March 2024","url":"https://themarkup.org/artificial-intelligence/2024/03/29/nycs-ai-chatbot-tells-businesses-to-break-the-law"}},{"text":"Finance worker pays out $25 million after video call with deepfake ‘chief financial officer’","answer":"Misuse","note":"The technology worked as intended. Someone pointed it at people.","source":{"title":"CNN, 4 February 2024","url":"https://edition.cnn.com/2024/02/04/asia/deepfake-cfo-scam-hong-kong-intl-hnk"}},{"text":"Millions of black people affected by racial bias in health-care algorithms","answer":"Design","note":"The model worked as built. The design is the problem: what it learned from, or what it was told to aim for.","source":{"title":"Nature, 24 October 2019","url":"https://www.nature.com/articles/d41586-019-03228-6"}},{"text":"FCC proposes $6 million fine for illegal robocalls that used Biden deepfake generative AI voice message","answer":"Misuse","note":"The technology worked as intended. Someone pointed it at people.","source":{"title":"Federal Communications Commission, news release, 23 May 2024","url":"https://docs.fcc.gov/public/attachments/DOC-402762A1.pdf"}},{"text":"Google restricts AI search tool after ‘nonsensical’ answers told people to eat rocks and put glue on pizza","answer":"Capability","note":"The system failed at the job it claims to do. It gave a plausible answer that was not true.","source":{"title":"Forbes, 31 May 2024","url":"https://www.forbes.com/sites/roberthart/2024/05/31/google-restricts-ai-search-tool-after-nonsensical-answers-told-people-to-eat-rocks-and-put-glue-on-pizza/"}}]},{"t":"h2","id":"three-bins-three-fixes","runs":[{"text":"Three bins, three fixes"}]},{"t":"p","runs":[{"text":"Design.","b":true},{"text":" The system works as built, and the build is the problem. Amazon’s experimental hiring tool learned from ten years of CVs, most of them from men, and marked down the word “women’s”. It learned what it was shown. An illustrative case makes the same point and runs through this course’s film: a bank’s loan model turns down a whole district, because few loans were ever made there, so few repayments were ever recorded. The fix is a different target, different data or a different threshold."}]},{"t":"p","runs":[{"text":"Capability.","b":true},{"text":" The system fails at the thing it claims to do. ChatGPT is not a legal database. It produces plausible text, and a plausible citation is what Steven Schwartz asked it for. The fix sits outside the tool: read every citation in a primary source before you sign."}]},{"t":"p","runs":[{"text":"Misuse.","b":true},{"text":" The system works, and somebody points it at you. In 2024 a finance worker in Hong Kong paid out about $25 million after a video call in which every colleague was a deepfake. The fix is procedure: hang up and call back on a number you already hold."}]},{"t":"h2","id":"find-the-person-who-chose","runs":[{"text":"Find the person who chose"}]},{"t":"p","runs":[{"text":"When a system turns you down, no machine decided. People chose the model, the data, the target, the score at which a yes becomes a no, and whether anyone looks again. “The algorithm decided” moves those choices from people to software."}]},{"t":"p","runs":[{"text":"In "},{"text":"Mata","i":true},{"text":" the people are easy to name. One lawyer used a tool he had never used before. Another signed the brief without reading the cases. When the airline said the cases could not be found, both stood by them. On 22 June 2023 Judge P. Kevin Castel fined them and their firm $5,000. He wrote that using a reliable AI tool is not improper in itself, and that the rules make lawyers the gatekeepers of what they file."}]},{"t":"p","runs":[{"text":"So ask two questions of any failure. Which bin is it in? Which human decision put it there? The whole Mata story, with the choice the firm faced in April 2023, is this module’s case study."}]}],"check":{"question":[{"text":"A delivery firm’s routing model sends vans past a primary school at pick-up time, because it was told to minimise minutes and nothing else. Which kind of failure is this?"}],"options":[{"text":[{"text":"Capability: the model failed to find the best route for vans"}],"correct":false,"tempting":[{"text":"The route looks like a mistake, so it feels as if the tool fell short. It found exactly the fastest route it was asked for."}]},{"text":[{"text":"Misuse: somebody aimed the routing tool at the schoolchildren"}],"correct":false,"tempting":[{"text":"The harm lands on children who never chose the tool, which feels like being targeted. Nobody pointed it at anyone."}]},{"text":[{"text":"Design: it met its target, and the target left out safety"}],"correct":true,"tempting":null},{"text":[{"text":"Capability: the map data must have been old or incomplete"}],"correct":false,"tempting":[{"text":"Old maps do cause bad routes. Nothing here says the map was wrong, and the route really was the fastest."}]}],"order":[1,0,2,3],"why":[{"text":"Minutes were the only thing it was told to value. The fix is a different target, and a person chooses it."}]},"prev":null,"next":{"slug":"what-the-machine-is-doing","title":"What the model computes","minutes":4}}}