The four numbers, worked through

One thousand transactions

A fraud model runs on 1,000 transactions. 40 are fraudulent. At a threshold of 0.5 the model flags 30 transactions, and 24 of them are fraud.

Four numbers describe everything:

True positives  (TP) = 24    flagged and fraudulent
False positives (FP) =  6    flagged, actually fine
False negatives (FN) = 16    not flagged, actually fraud
True negatives  (TN) = 954   not flagged, fine

Check the arithmetic: 24 + 6 + 16 + 954 = 1,000, and 24 + 16 = 40 actual fraud cases. Everything below is derived from these four.

The metrics, computed

Accuracy = (TP + TN) / all = 978 / 1000 = 97.8%. And the model that flags nothing scores 96%. Two points of improvement for all that work: this is the accuracy trap from the reuse lesson, in numbers.

Precision = TP / (TP + FP) = 24 / 30 = 80%. Of what we flagged, 80% was fraud. This is the number the investigation team cares about, because it determines how much of their day is wasted.

Recall = TP / (TP + FN) = 24 / 40 = 60%. Of all real fraud, we caught 60%. This is the number the finance team cares about, because 40% of the losses walked past.

F1 = harmonic mean of precision and recall = 2 x (0.8 x 0.6) / (0.8 + 0.6) = 0.686. The harmonic mean rather than the arithmetic one, because it punishes imbalance: a model with precision 1.0 and recall 0.02 has arithmetic mean 0.51 and F1 of 0.039. F1 refuses to be impressed by one number when the other is terrible.

Specificity = TN / (TN + FP) = 954 / 960 = 99.4%. Of the innocent, how many we left alone. Standard in medicine, rarer in industry, and the complement of the false positive rate.

The two directions people confuse

Precision and recall are both about the positive class but they divide by different things, and mixing them up is the most common error in this area.

• Precision divides by what the model said. Column of the matrix.
• Recall divides by what was true. Row of the matrix.

A sentence that fixes it permanently: precision is how much of my alarm was justified; recall is how much of the danger I noticed.

Where each one matters

Recall matters most when a miss is catastrophic and a false alarm is merely annoying. Cancer screening: a missed tumour is fatal, a false positive is an unpleasant follow-up scan. Safety alerts. Content moderation for illegal material.

Precision matters most when acting on a positive is expensive or harmful. Sending a debt collector. Freezing an account. Any automated action against a person. If precision is 20%, four out of every five people you accuse are innocent, and that is a decision about people, not a metric.

Both, at a stated operating point, is the honest report. “At a threshold of 0.62 we get 71% precision and 55% recall” tells someone what will actually happen. “F1 of 0.62” does not, and it hides which side of the trade-off you chose.

The number that is not on this list

Most people asked “the model flagged this transaction: what is the chance it is really fraud?” reach for recall or the model’s accuracy. Neither answers it. The answer is precision, and precision depends on the base rate in a way that catches almost everyone.

Run the same model, 60% recall, 99.4% specificity, on a population where fraud is 0.1% rather than 4%. Out of 100,000 transactions, 100 are fraud. The model catches 60. It also flags 0.6% of the 99,900 clean ones, which is 599 false positives. Precision is 60 / 659 = 9%.

Same model, same sensitivity, same specificity. Precision fell from 80% to 9% because the base rate fell. This is the base rate fallacy, it is why a rare-disease screening test with excellent sensitivity still produces mostly false positives, and it is the single most important thing on this page.

So whenever someone quotes a precision figure, ask what the prevalence was in the set it was measured on. If your production prevalence differs, the precision you were promised does not transfer.

Getting it out of the library

from sklearn.metrics import confusion_matrix, classification_report
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
print(classification_report(y_true, y_pred, digits=3))

Note the order that ravel() gives you (tn, fp, fn, tp) which is not the order most people expect and is worth checking the first few times rather than assuming.