In March 2023 two New York lawyers filed a brief in Mata v. Avianca, a personal injury claim against an airline in the federal court in Manhattan. Six of the decisions it cited did not exist. ChatGPT had written them, with judges’ names, docket numbers and quotations. When one lawyer asked it whether a case was real, it said yes.
That is one failure. The news carries dozens a week, and they are not all the same kind. Before reading on, sort six real headlines into three bins.
Try it
Six real headlines. Put each one in the bin where its failure comes from.
- Design: it worked as built, and the design is the problem.
- Capability: it failed at the job it claims to do.
- Misuse: it worked, and someone aimed it at people.
Amazon scraps secret AI recruiting tool that showed bias against women
NYC’s AI chatbot tells businesses to break the law
Finance worker pays out $25 million after video call with deepfake ‘chief financial officer’
Millions of black people affected by racial bias in health-care algorithms
FCC proposes $6 million fine for illegal robocalls that used Biden deepfake generative AI voice message
Federal Communications Commission, news release, 23 May 2024
Google restricts AI search tool after ‘nonsensical’ answers told people to eat rocks and put glue on pizza
Three bins, three fixes
Design. The system works as built, and the build is the problem. Amazon’s experimental hiring tool learned from ten years of CVs, most of them from men, and marked down the word “women’s”. It learned what it was shown. An illustrative case makes the same point and runs through this course’s film: a bank’s loan model turns down a whole district, because few loans were ever made there, so few repayments were ever recorded. The fix is a different target, different data or a different threshold.
Capability. The system fails at the thing it claims to do. ChatGPT is not a legal database. It produces plausible text, and a plausible citation is what Steven Schwartz asked it for. The fix sits outside the tool: read every citation in a primary source before you sign.
Misuse. The system works, and somebody points it at you. In 2024 a finance worker in Hong Kong paid out about $25 million after a video call in which every colleague was a deepfake. The fix is procedure: hang up and call back on a number you already hold.
Find the person who chose
When a system turns you down, no machine decided. People chose the model, the data, the target, the score at which a yes becomes a no, and whether anyone looks again. “The algorithm decided” moves those choices from people to software.
In Mata the people are easy to name. One lawyer used a tool he had never used before. Another signed the brief without reading the cases. When the airline said the cases could not be found, both stood by them. On 22 June 2023 Judge P. Kevin Castel fined them and their firm $5,000. He wrote that using a reliable AI tool is not improper in itself, and that the rules make lawyers the gatekeepers of what they file.
So ask two questions of any failure. Which bin is it in? Which human decision put it there? The whole Mata story, with the choice the firm faced in April 2023, is this module’s case study.
Check your understanding
A delivery firm’s routing model sends vans past a primary school at pick-up time, because it was told to minimise minutes and nothing else. Which kind of failure is this?
Pick the one you would defend. Your answer stays private.
Stuck on something?Ask Rukmini, your AI tutor