AI, Safety and What Goes Wrong · Chapter 01

Three kinds of failure

2:47 · Narrated with a synthetic voice. Captions are available in the player.

Read the transcript

In March twenty twenty-three, two New York lawyers filed a brief, and six of the court decisions it cited did not exist. ChatGPT had written them, with judges’ names and quotations, and when one lawyer asked whether a case was real, it said yes. That is a failure. So is this one, and it runs through the whole of this film. A bank builds a model to predict who will repay a loan. It works. It also turns down almost everyone from one district of the city, because few loans were ever made there, so few repayments were ever recorded. This bank is an illustration, built from a pattern that is well documented. The two failures are different kinds. The lawyers’ tool failed at the thing they believed it did. That is capability. The loan model does exactly what it was built to do, and the build is the problem. That is design. The third kind is misuse: the tool works, and somebody points it at you, like a cloned voice of a president, telling voters to stay at home. Each kind needs a different fix. So if nothing in the loan model is broken, who decided to turn that district down? People did. Somebody chose to build the model. Somebody chose the data: years of the bank’s own lending, in which that district barely appears. Somebody chose the target, repayment, and the score at which a yes becomes a no. And somebody chose whether a person looks again before the letter goes out. Every one of those decisions has a name attached. The phrase, the algorithm decided, moves them from people to software, and it usually appears in a press statement. The court in the lawyers’ case did not accept that move. Asked whether its case was real, the chatbot had said yes, and that it could be found on Westlaw and LexisNexis. The judge found its legal analysis was gibberish. On the twenty-second of June, twenty twenty-three he fined the two lawyers and their firm five thousand dollars, and wrote that the rules make lawyers the gatekeepers of what they file. A tool can write the text. A person still signs it. So, back to the bank. How does a model that works exactly as built learn to avoid a whole district?

Sources and further reading (1)