Precision and recall, worked on real numbers

Four cells, and everything comes from them

Any yes/no classifier produces four outcomes on a labelled set. A content filter run over 1,000 posts, of which 60 are harmful:

• It flagged 80 posts.
• Of those 80, 48 were harmful. These are true positives.
• The other 32 flagged posts were fine. False positives.
• 12 harmful posts went through unflagged. False negatives.
• 908 fine posts were correctly left alone. True negatives.

That is the confusion matrix. Every metric in this lesson is arithmetic on those four numbers, and you should be able to do all of it on paper.

Precision: when it flags, how often is it right?

precision = TP / (TP + FP) = 48 / 80 = 0.60

Recall: of the things that should have been caught, how many were?

recall = TP / (TP + FN) = 48 / 60 = 0.80

F1: the harmonic mean of the two, used when you want one number:

F1 = 2 * P * R / (P + R) = 2 * 0.6 * 0.8 / 1.4 = 0.686

Accuracy is the trap

Accuracy is (TP + TN) / total = (48 + 908) / 1000 = 95.6%. It sounds like a good system.

Now consider a system that flags nothing at all. It has zero true positives, zero false positives, and 940 true negatives. Its accuracy is 94.0%.

A classifier that does literally nothing scores within 1.6 points of the working one. That is what accuracy does under class imbalance, and imbalance is the normal case: fraud, spam, defects, safety violations, disease, churn. The rarer the thing you care about, the more accuracy measures your ability to say “no”.

Whenever somebody reports accuracy on a rare event, ask for the base rate. If they do not know it, they do not have a result.

Which of the two you care about is a business question

Precision and recall trade against each other and nothing in the mathematics tells you where to sit. The product does.

• Recall matters more when a miss is the expensive failure and a false alarm is merely annoying: screening for a serious disease, catching child-safety violations, detecting a payment outage. You accept reviewing extra cases.
• Precision matters more when acting on a positive is costly or irreversible and misses are recoverable: auto-deleting a customer’s post, auto-refunding money, sending an accusatory email. Here, a wrong action costs trust you cannot buy back.
• Neither, alone, when a human is downstream. If flagged items go to a review queue, the real constraint is the queue’s capacity. Precision then translates directly into wasted reviewer minutes, and the sane metric is “harmful posts caught per reviewer-hour”.

F1 hides this choice by averaging it away. Report precision and recall as a pair and quote F1 only as a summary, never as the thing you optimise without saying which side you would rather err on.

More than two classes

With several categories (say routing tickets to billing, technical, sales, other) you compute precision and recall per class, then average. How you average changes the story:

• Macro average treats every class equally. A class with 8 examples counts as much as one with 800. Use it when the rare classes matter.
• Micro average weights by frequency, so it is dominated by the big classes and equals accuracy in the single-label case.
• Weighted average sits in between.

A router can have a macro-F1 of 0.52 and a micro-F1 of 0.88 at the same time, because it handles the two common categories well and the three rare ones badly. Both numbers are true. Only one of them tells you the sales queue is broken.

Always print the per-class breakdown before any average. In scikit-learn that is one call:

from sklearn.metrics import classification_report, confusion_matrix
print(confusion_matrix(y_true, y_pred, labels=labels))
print(classification_report(y_true, y_pred, labels=labels, digits=3))

Both are free and run on a laptop; if you have no Python, pandas.crosstab or even a pivot table in LibreOffice Calc gives you the same four cells.

Read the matrix, not just the metrics

The cells themselves tell you what to fix. Print the full grid and look at where the mass sits off the diagonal. “Technical” being misread as “billing” 40 times is one specific, fixable confusion, often two categories that a human would also struggle to separate, which is a taxonomy problem, not a model problem.

This is the fastest route from a disappointing number to an actual next action, and it takes about ten minutes. A summary metric never tells you which two classes to merge.