AI, Safety and What Goes Wrong · Chapter 02

Four boxes, counted by group

2:45 · Narrated with a synthetic voice. Captions are available in the player.

Read the transcript

It learns from outcomes, and you can see what it learned by counting. Every loan decision lands in one of four boxes: approved and repaid, approved and defaulted, refused but would have repaid, and refused and would have defaulted. Now count the boxes separately for each group of applicants. The course works an example with numbers chosen to divide cleanly: a thousand applicants from each of two groups. Among the people in group B who would have repaid, the model approves seventy per cent. In group A, it approves eighty-seven and a half per cent. Yet of the loans it does approve, eighty-seven and a half per cent are repaid in group B, and ninety per cent in group A. So the bank can say, truthfully, that an approval means much the same thing for everyone. And an applicant from group B who deserves a loan is refused thirty per cent of the time, against twelve and a half per cent in group A. Both statements come from the same four boxes. A single accuracy figure hides the argument between them. So why not simply delete the column that says which group an applicant is in? Because other columns rebuild it. Delete the group, and the postcode is still there. In our bank, the postcode is the district. A model forbidden to use one learns the same pattern from the other, and now you have lost the column you needed to measure the gap. There is a second trap, and it is quieter. A lender that approves someone later learns whether they repaid. A lender that refuses someone learns nothing, ever. So the training data holds outcomes only for the people who were approved. Almost nobody from the district was approved, so the district produces almost no evidence that it would have repaid. The next model trains on that silence, refuses the district again, and grows more confident, because the data seems to agree. This is called the selective labels problem, and it turns one bad decision into a policy. The model sees no error, so it reports none. The only way out is to go and look: approve some applicants the model would refuse, and see who repays. So when a refused applicant asks the bank’s chat assistant why, what does it tell them?

Sources and further reading (1)