Machine Learning, Foundations · Chapter 05
The four numbers
1:59 · Narrated with a synthetic voice. Captions are available in the player.
Read the transcript
A thousand transactions. Forty of them are fraud. The model flags thirty and twenty-four of those are real. Accuracy is ninety-seven point eight per cent, and the model that flags nothing scores ninety-six, so accuracy has told you almost nothing. Precision is twenty-four divided by thirty, eighty per cent: of what we flagged, this much was real. Recall is twenty-four divided by forty, sixty per cent: of what was really there, this much was caught. Precision divides by what the model said. Recall divides by what was true. And precision is not a property of the model. Run the same model where fraud is one in a thousand instead of one in twenty-five, and precision falls from eighty per cent to nine. The model produces a score. Something else turns that score into a decision, and that something is a threshold you chose. Zero point five is a default somebody typed into a library. It is correct only when the two kinds of error cost the same and the classes are balanced, which describes almost nothing. If a missed fraud costs eighteen thousand rupees and a review costs ninety, the break-even threshold is ninety divided by eighteen thousand and ninety, which is about half of one per cent. Or the constraint is capacity rather than cost. The team can review two hundred cases a day, so the threshold is whatever produces two hundred flags, and the metric is precision among those two hundred.