{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"the-confusion-matrix","title":"Precision and recall, worked on real numbers","url":"https://addaly.com/learn/evaluating-ai/the-confusion-matrix","position":12,"of":76,"minutes":9,"answer":"Precision, recall and accuracy come from the same four cells, and under class imbalance accuracy rewards a system that does nothing.","keyIdea":"Precision, recall and accuracy come from the same four cells, and under class imbalance accuracy rewards a system that does nothing."},"body":{"format":"markdown","text":"## Four cells, and everything comes from them\n\nAny yes/no classifier produces four outcomes on a labelled set. A content filter run over 1,000 posts, of which 60 are harmful:\n\n- It flagged 80 posts.\n- Of those 80, 48 were harmful. These are **true positives**.\n- The other 32 flagged posts were fine. **False positives**.\n- 12 harmful posts went through unflagged. **False negatives**.\n- 908 fine posts were correctly left alone. **True negatives**.\n\nThat is the confusion matrix. Every metric in this lesson is arithmetic on those four numbers, and you should be able to do all of it on paper.\n\n> Figure: A content filter over 1,000 posts, 60 of them harmful. A two by two grid, rows Actually harmful and Actually fine, columns Flagged and Not flagged: Actually harmful and Flagged: 48; Actually harmful and Not flagged: 12; Actually fine and Flagged: 32; Actually fine and Not flagged: 908. Precision 48/80 = 0.60, recall 48/60 = 0.80, accuracy 95.6%. A filter that flags nothing at all scores 94.0% accuracy from the bottom-right cell alone.\n\n**Precision**: when it flags, how often is it right?\n\n```\nprecision = TP / (TP + FP) = 48 / 80 = 0.60\n```\n\n**Recall**: of the things that should have been caught, how many were?\n\n```\nrecall = TP / (TP + FN) = 48 / 60 = 0.80\n```\n\n**F1**: the harmonic mean of the two, used when you want one number:\n\n```\nF1 = 2 * P * R / (P + R) = 2 * 0.6 * 0.8 / 1.4 = 0.686\n```\n\n## Accuracy is the trap\n\nAccuracy is `(TP + TN) / total` = `(48 + 908) / 1000` = **95.6%**. It sounds like a good system.\n\nNow consider a system that flags nothing at all. It has zero true positives, zero false positives, and 940 true negatives. Its accuracy is **94.0%**.\n\nA classifier that does literally nothing scores within 1.6 points of the working one. That is what accuracy does under class imbalance, and imbalance is the normal case: fraud, spam, defects, safety violations, disease, churn. The rarer the thing you care about, the more accuracy measures your ability to say \"no\".\n\nWhenever somebody reports accuracy on a rare event, ask for the base rate. If they do not know it, they do not have a result.\n\n## Which of the two you care about is a business question\n\nPrecision and recall trade against each other and nothing in the mathematics tells you where to sit. The product does.\n\n- **Recall matters more** when a miss is the expensive failure and a false alarm is merely annoying: screening for a serious disease, catching child-safety violations, detecting a payment outage. You accept reviewing extra cases.\n- **Precision matters more** when acting on a positive is costly or irreversible and misses are recoverable: auto-deleting a customer's post, auto-refunding money, sending an accusatory email. Here, a wrong action costs trust you cannot buy back.\n- **Neither, alone, when a human is downstream.** If flagged items go to a review queue, the real constraint is the queue's capacity. Precision then translates directly into wasted reviewer minutes, and the sane metric is \"harmful posts caught per reviewer-hour\".\n\nF1 hides this choice by averaging it away. Report precision and recall as a pair and quote F1 only as a summary, never as the thing you optimise without saying which side you would rather err on.\n\n## More than two classes\n\nWith several categories (say routing tickets to billing, technical, sales, other) you compute precision and recall per class, then average. How you average changes the story:\n\n- **Macro average** treats every class equally. A class with 8 examples counts as much as one with 800. Use it when the rare classes matter.\n- **Micro average** weights by frequency, so it is dominated by the big classes and equals accuracy in the single-label case.\n- **Weighted average** sits in between.\n\nA router can have a macro-F1 of 0.52 and a micro-F1 of 0.88 at the same time, because it handles the two common categories well and the three rare ones badly. Both numbers are true. Only one of them tells you the sales queue is broken.\n\nAlways print the per-class breakdown before any average. In scikit-learn that is one call:\n\n```python\nfrom sklearn.metrics import classification_report, confusion_matrix\nprint(confusion_matrix(y_true, y_pred, labels=labels))\nprint(classification_report(y_true, y_pred, labels=labels, digits=3))\n```\n\nBoth are free and run on a laptop; if you have no Python, `pandas.crosstab` or even a pivot table in LibreOffice Calc gives you the same four cells.\n\n## Read the matrix, not just the metrics\n\nThe cells themselves tell you what to fix. Print the full grid and look at where the mass sits off the diagonal. \"Technical\" being misread as \"billing\" 40 times is one specific, fixable confusion, often two categories that a human would also struggle to separate, which is a taxonomy problem, not a model problem.\n\nThis is the fastest route from a disappointing number to an actual next action, and it takes about ten minutes. A summary metric never tells you which two classes to merge."},"check":{"question":"A defect detector runs over 5,000 parts, 100 of which are genuinely defective. It flags 60 parts and 45 of those are truly defective. Which statement is correct?","options":[{"text":"With 98.8% accuracy the detector is performing well and is ready to run unattended.","correct":false,"tempting":"Quotes accuracy on a task where 98% of items are fine, so a detector that flagged nothing would score 98% too."},{"text":"Recall is 45%, so more than half the defective parts are shipping, even though precision is a respectable 75%.","correct":true},{"text":"Precision is 45% and recall is 75%, so the main cost is wasted inspection time.","correct":false,"tempting":"Swaps the two formulas: dividing by the number flagged gives precision, dividing by the number actually defective gives recall."},{"text":"F1 of 0.56 shows the detector is close to random and the approach should be abandoned.","correct":false,"tempting":"Reads an F1 in the middle of the range as near-random when random flagging at this base rate would give a precision near 2%."}]},"prev":{"slug":"measuring-text","title":"What to measure when the output is prose","url":"https://addaly.com/learn/evaluating-ai/measuring-text"},"next":{"slug":"multi-label-and-partial-credit","title":"When an item has several right answers","url":"https://addaly.com/learn/evaluating-ai/multi-label-and-partial-credit"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Precision and recall, worked on real numbers', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/the-confusion-matrix (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Precision and recall, worked on real numbers","courseTitle":"Knowing If It Works","module":{"index":2,"slug":"metrics-that-mean-something","title":"Metrics that mean something"},"answer":null,"keyIdea":"Precision, recall and accuracy come from the same four cells, and under class imbalance accuracy rewards a system that does nothing.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["Flagged","Not flagged"],"rows":["Actually harmful","Actually fine"],"type":"matrix","cells":[["48","12"],["32","908"]],"title":"A content filter over 1,000 posts, 60 of them harmful","caption":"Precision 48/80 = 0.60, recall 48/60 = 0.80, accuracy 95.6%. A filter that flags nothing at all scores 94.0% accuracy from the bottom-right cell alone.","cellLabels":[["true positives","false negatives"],["false positives","true negatives"]]},"alt":"Figure: A content filter over 1,000 posts, 60 of them harmful. A two by two grid, rows Actually harmful and Actually fine, columns Flagged and Not flagged: Actually harmful and Flagged: 48; Actually harmful and Not flagged: 12; Actually fine and Flagged: 32; Actually fine and Not flagged: 908. Precision 48/80 = 0.60, recall 48/60 = 0.80, accuracy 95.6%. A filter that flags nothing at all scores 94.0% accuracy from the bottom-right cell alone."}},"blocks":[{"t":"h2","id":"four-cells-and-everything-comes-from-them","runs":[{"text":"Four cells, and everything comes from them"}]},{"t":"p","runs":[{"text":"Any yes/no classifier produces four outcomes on a labelled set. A content filter run over 1,000 posts, of which 60 are harmful:"}]},{"t":"ul","items":[[{"text":"It flagged 80 posts."}],[{"text":"Of those 80, 48 were harmful. These are "},{"text":"true positives","b":true},{"text":"."}],[{"text":"The other 32 flagged posts were fine. "},{"text":"False positives","b":true},{"text":"."}],[{"text":"12 harmful posts went through unflagged. "},{"text":"False negatives","b":true},{"text":"."}],[{"text":"908 fine posts were correctly left alone. "},{"text":"True negatives","b":true},{"text":"."}]]},{"t":"p","runs":[{"text":"That is the confusion matrix. Every metric in this lesson is arithmetic on those four numbers, and you should be able to do all of it on paper."}]},{"t":"p","runs":[{"text":"Precision","b":true},{"text":": when it flags, how often is it right?"}]},{"t":"code","lang":"","label":"Text","text":"precision = TP / (TP + FP) = 48 / 80 = 0.60"},{"t":"p","runs":[{"text":"Recall","b":true},{"text":": of the things that should have been caught, how many were?"}]},{"t":"code","lang":"","label":"Text","text":"recall = TP / (TP + FN) = 48 / 60 = 0.80"},{"t":"p","runs":[{"text":"F1","b":true},{"text":": the harmonic mean of the two, used when you want one number:"}]},{"t":"code","lang":"","label":"Text","text":"F1 = 2 * P * R / (P + R) = 2 * 0.6 * 0.8 / 1.4 = 0.686"},{"t":"h2","id":"accuracy-is-the-trap","runs":[{"text":"Accuracy is the trap"}]},{"t":"p","runs":[{"text":"Accuracy is "},{"text":"(TP + TN) / total","code":true},{"text":" = "},{"text":"(48 + 908) / 1000","code":true},{"text":" = "},{"text":"95.6%","b":true},{"text":". It sounds like a good system."}]},{"t":"p","runs":[{"text":"Now consider a system that flags nothing at all. It has zero true positives, zero false positives, and 940 true negatives. Its accuracy is "},{"text":"94.0%","b":true},{"text":"."}]},{"t":"p","runs":[{"text":"A classifier that does literally nothing scores within 1.6 points of the working one. That is what accuracy does under class imbalance, and imbalance is the normal case: fraud, spam, defects, safety violations, disease, churn. The rarer the thing you care about, the more accuracy measures your ability to say “no”."}]},{"t":"p","runs":[{"text":"Whenever somebody reports accuracy on a rare event, ask for the base rate. If they do not know it, they do not have a result."}]},{"t":"h2","id":"which-of-the-two-you-care-about-is-a-business-question","runs":[{"text":"Which of the two you care about is a business question"}]},{"t":"p","runs":[{"text":"Precision and recall trade against each other and nothing in the mathematics tells you where to sit. The product does."}]},{"t":"ul","items":[[{"text":"Recall matters more","b":true},{"text":" when a miss is the expensive failure and a false alarm is merely annoying: screening for a serious disease, catching child-safety violations, detecting a payment outage. You accept reviewing extra cases."}],[{"text":"Precision matters more","b":true},{"text":" when acting on a positive is costly or irreversible and misses are recoverable: auto-deleting a customer’s post, auto-refunding money, sending an accusatory email. Here, a wrong action costs trust you cannot buy back."}],[{"text":"Neither, alone, when a human is downstream.","b":true},{"text":" If flagged items go to a review queue, the real constraint is the queue’s capacity. Precision then translates directly into wasted reviewer minutes, and the sane metric is “harmful posts caught per reviewer-hour”."}]]},{"t":"p","runs":[{"text":"F1 hides this choice by averaging it away. Report precision and recall as a pair and quote F1 only as a summary, never as the thing you optimise without saying which side you would rather err on."}]},{"t":"h2","id":"more-than-two-classes","runs":[{"text":"More than two classes"}]},{"t":"p","runs":[{"text":"With several categories (say routing tickets to billing, technical, sales, other) you compute precision and recall per class, then average. How you average changes the story:"}]},{"t":"ul","items":[[{"text":"Macro average","b":true},{"text":" treats every class equally. A class with 8 examples counts as much as one with 800. Use it when the rare classes matter."}],[{"text":"Micro average","b":true},{"text":" weights by frequency, so it is dominated by the big classes and equals accuracy in the single-label case."}],[{"text":"Weighted average","b":true},{"text":" sits in between."}]]},{"t":"p","runs":[{"text":"A router can have a macro-F1 of 0.52 and a micro-F1 of 0.88 at the same time, because it handles the two common categories well and the three rare ones badly. Both numbers are true. Only one of them tells you the sales queue is broken."}]},{"t":"p","runs":[{"text":"Always print the per-class breakdown before any average. In scikit-learn that is one call:"}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.metrics import classification_report, confusion_matrix\nprint(confusion_matrix(y_true, y_pred, labels=labels))\nprint(classification_report(y_true, y_pred, labels=labels, digits=3))"},{"t":"p","runs":[{"text":"Both are free and run on a laptop; if you have no Python, "},{"text":"pandas.crosstab","code":true},{"text":" or even a pivot table in LibreOffice Calc gives you the same four cells."}]},{"t":"h2","id":"read-the-matrix-not-just-the-metrics","runs":[{"text":"Read the matrix, not just the metrics"}]},{"t":"p","runs":[{"text":"The cells themselves tell you what to fix. Print the full grid and look at where the mass sits off the diagonal. “Technical” being misread as “billing” 40 times is one specific, fixable confusion, often two categories that a human would also struggle to separate, which is a taxonomy problem, not a model problem."}]},{"t":"p","runs":[{"text":"This is the fastest route from a disappointing number to an actual next action, and it takes about ten minutes. A summary metric never tells you which two classes to merge."}]}],"check":{"question":[{"text":"A defect detector runs over 5,000 parts, 100 of which are genuinely defective. It flags 60 parts and 45 of those are truly defective. Which statement is correct?"}],"options":[{"text":[{"text":"With 98.8% accuracy the detector is performing well and is ready to run unattended."}],"correct":false,"tempting":[{"text":"Quotes accuracy on a task where 98% of items are fine, so a detector that flagged nothing would score 98% too."}]},{"text":[{"text":"Recall is 45%, so more than half the defective parts are shipping, even though precision is a respectable 75%."}],"correct":true,"tempting":null},{"text":[{"text":"Precision is 45% and recall is 75%, so the main cost is wasted inspection time."}],"correct":false,"tempting":[{"text":"Swaps the two formulas: dividing by the number flagged gives precision, dividing by the number actually defective gives recall."}]},{"text":[{"text":"F1 of 0.56 shows the detector is close to random and the approach should be abandoned."}],"correct":false,"tempting":[{"text":"Reads an F1 in the middle of the range as near-random when random flagging at this base rate would give a precision near 2%."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"measuring-text","title":"What to measure when the output is prose","minutes":8},"next":{"slug":"multi-label-and-partial-credit","title":"When an item has several right answers","minutes":8}}}