{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"confusion-matrix-worked","title":"The four numbers, worked through","url":"https://addaly.com/learn/machine-learning-foundations/confusion-matrix-worked","position":43,"of":92,"minutes":9,"answer":"Precision divides by what the model said and recall by what was true, and precision alone is not a property of the model: it moves with the base rate of the population you run on.","keyIdea":"Precision divides by what the model said and recall by what was true, and precision alone is not a property of the model: it moves with the base rate of the population you run on."},"body":{"format":"markdown","text":"## One thousand transactions\n\nA fraud model runs on 1,000 transactions. 40 are fraudulent. At a threshold of 0.5 the model flags 30 transactions, and 24 of them are fraud.\n\nFour numbers describe everything:\n\n```text\nTrue positives  (TP) = 24    flagged and fraudulent\nFalse positives (FP) =  6    flagged, actually fine\nFalse negatives (FN) = 16    not flagged, actually fraud\nTrue negatives  (TN) = 954   not flagged, fine\n```\n\nCheck the arithmetic: 24 + 6 + 16 + 954 = 1,000, and 24 + 16 = 40 actual fraud cases. Everything below is derived from these four.\n\n> Figure: 1,000 transactions, 40 fraudulent, threshold 0.5. A two by two grid, rows Actually fraud and Actually fine, columns Flagged and Not flagged: Actually fraud and Flagged: 24; Actually fraud and Not flagged: 16; Actually fine and Flagged: 6; Actually fine and Not flagged: 954. Precision 24 / 30 = 80 per cent divides by what the model said: the Flagged column. Recall 24 / 40 = 60 per cent divides by what was true: the top row. Accuracy is 97.8 per cent, against 96 per cent for a model that flags nothing at all.\n\n## The metrics, computed\n\n**Accuracy** = (TP + TN) / all = 978 / 1000 = **97.8%**. And the model that flags nothing scores 96%. Two points of improvement for all that work: this is the accuracy trap from the reuse lesson, in numbers.\n\n**Precision** = TP / (TP + FP) = 24 / 30 = **80%**. Of what we flagged, 80% was fraud. This is the number the investigation team cares about, because it determines how much of their day is wasted.\n\n**Recall** = TP / (TP + FN) = 24 / 40 = **60%**. Of all real fraud, we caught 60%. This is the number the finance team cares about, because 40% of the losses walked past.\n\n**F1** = harmonic mean of precision and recall = 2 x (0.8 x 0.6) / (0.8 + 0.6) = **0.686**. The harmonic mean rather than the arithmetic one, because it punishes imbalance: a model with precision 1.0 and recall 0.02 has arithmetic mean 0.51 and F1 of 0.039. F1 refuses to be impressed by one number when the other is terrible.\n\n**Specificity** = TN / (TN + FP) = 954 / 960 = **99.4%**. Of the innocent, how many we left alone. Standard in medicine, rarer in industry, and the complement of the false positive rate.\n\n## The two directions people confuse\n\nPrecision and recall are both about the positive class but they divide by different things, and mixing them up is the most common error in this area.\n\n- Precision divides by **what the model said**. Column of the matrix.\n- Recall divides by **what was true**. Row of the matrix.\n\nA sentence that fixes it permanently: *precision is how much of my alarm was justified; recall is how much of the danger I noticed.*\n\n## Where each one matters\n\n**Recall matters most** when a miss is catastrophic and a false alarm is merely annoying. Cancer screening: a missed tumour is fatal, a false positive is an unpleasant follow-up scan. Safety alerts. Content moderation for illegal material.\n\n**Precision matters most** when acting on a positive is expensive or harmful. Sending a debt collector. Freezing an account. Any automated action against a person. If precision is 20%, four out of every five people you accuse are innocent, and that is a decision about people, not a metric.\n\n**Both, at a stated operating point**, is the honest report. \"At a threshold of 0.62 we get 71% precision and 55% recall\" tells someone what will actually happen. \"F1 of 0.62\" does not, and it hides which side of the trade-off you chose.\n\n## The number that is not on this list\n\nMost people asked \"the model flagged this transaction: what is the chance it is really fraud?\" reach for recall or the model's accuracy. Neither answers it. The answer is precision, and precision depends on the base rate in a way that catches almost everyone.\n\nRun the same model, 60% recall, 99.4% specificity, on a population where fraud is 0.1% rather than 4%. Out of 100,000 transactions, 100 are fraud. The model catches 60. It also flags 0.6% of the 99,900 clean ones, which is 599 false positives. Precision is 60 / 659 = **9%**.\n\nSame model, same sensitivity, same specificity. Precision fell from 80% to 9% because the base rate fell. This is the base rate fallacy, it is why a rare-disease screening test with excellent sensitivity still produces mostly false positives, and it is the single most important thing on this page.\n\n> Figure: The same model where fraud is 0.1 per cent of 100,000 transactions. A two by two grid, rows Actually fraud and Actually fine, columns Flagged and Not flagged: Actually fraud and Flagged: 60; Actually fraud and Not flagged: 40; Actually fine and Flagged: 599; Actually fine and Not flagged: 99,301. Recall is still 60 per cent and specificity still 99.4 per cent. Precision is now 60 / 659 = 9 per cent, because 0.6 per cent of 99,900 clean transactions is 599 false alarms. Precision belongs to the population the model runs on, not to the model.\n\nSo whenever someone quotes a precision figure, ask what the prevalence was in the set it was measured on. If your production prevalence differs, the precision you were promised does not transfer.\n\n## Getting it out of the library\n\n```python\nfrom sklearn.metrics import confusion_matrix, classification_report\ntn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()\nprint(classification_report(y_true, y_pred, digits=3))\n```\n\nNote the order that `ravel()` gives you (`tn, fp, fn, tp`) which is not the order most people expect and is worth checking the first few times rather than assuming."},"check":{"question":"A screening model with 90% sensitivity and 95% specificity is moved from a high-risk clinic (10% prevalence) to general population screening (0.5% prevalence). What happens to the chance that a positive result is a true case?","options":[{"text":"It falls sharply, from about 67% to about 8%, because the far larger healthy population generates many more false positives even at an unchanged 5% false positive rate.","correct":true},{"text":"It stays the same, since sensitivity and specificity are properties of the test and do not change with population.","correct":false,"tempting":"Correctly notes that sensitivity and specificity are population-independent, but the probability a positive is correct is not: it depends on prevalence."},{"text":"It rises, because a lower prevalence means fewer diseased people to miss, so the positives that do appear are more reliable.","correct":false,"tempting":"Reverses the arithmetic: fewer true cases means the same false positive rate produces a larger share of the positive results."},{"text":"It cannot be determined without knowing the model's accuracy on the new population.","correct":false,"tempting":"Treats the calculation as under-determined when sensitivity, specificity and prevalence are sufficient to compute it exactly."}]},"prev":{"slug":"when-accuracy-lies","title":"When accuracy lies","url":"https://addaly.com/learn/machine-learning-foundations/when-accuracy-lies"},"next":{"slug":"choosing-the-threshold","title":"The threshold is a business decision, not a default","url":"https://addaly.com/learn/machine-learning-foundations/choosing-the-threshold"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The four numbers, worked through', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/confusion-matrix-worked (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The four numbers, worked through","courseTitle":"Machine Learning, Foundations","module":{"index":5,"slug":"measuring-honestly","title":"Measuring a model honestly"},"answer":null,"keyIdea":"Precision divides by what the model said and recall by what was true, and precision alone is not a property of the model: it moves with the base rate of the population you run on.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["Flagged","Not flagged"],"rows":["Actually fraud","Actually fine"],"type":"matrix","cells":[["24","16"],["6","954"]],"title":"1,000 transactions, 40 fraudulent, threshold 0.5","caption":"Precision 24 / 30 = 80 per cent divides by what the model said: the Flagged column. Recall 24 / 40 = 60 per cent divides by what was true: the top row. Accuracy is 97.8 per cent, against 96 per cent for a model that flags nothing at all.","cellLabels":[["true positives","false negatives"],["false positives","true negatives"]]},"alt":"Figure: 1,000 transactions, 40 fraudulent, threshold 0.5. A two by two grid, rows Actually fraud and Actually fine, columns Flagged and Not flagged: Actually fraud and Flagged: 24; Actually fraud and Not flagged: 16; Actually fine and Flagged: 6; Actually fine and Not flagged: 954. Precision 24 / 30 = 80 per cent divides by what the model said: the Flagged column. Recall 24 / 40 = 60 per cent divides by what was true: the top row. Accuracy is 97.8 per cent, against 96 per cent for a model that flags nothing at all."}},"blocks":[{"t":"h2","id":"one-thousand-transactions","runs":[{"text":"One thousand transactions"}]},{"t":"p","runs":[{"text":"A fraud model runs on 1,000 transactions. 40 are fraudulent. At a threshold of 0.5 the model flags 30 transactions, and 24 of them are fraud."}]},{"t":"p","runs":[{"text":"Four numbers describe everything:"}]},{"t":"code","lang":"text","label":"Text","text":"True positives  (TP) = 24    flagged and fraudulent\nFalse positives (FP) =  6    flagged, actually fine\nFalse negatives (FN) = 16    not flagged, actually fraud\nTrue negatives  (TN) = 954   not flagged, fine"},{"t":"p","runs":[{"text":"Check the arithmetic: 24 + 6 + 16 + 954 = 1,000, and 24 + 16 = 40 actual fraud cases. Everything below is derived from these four."}]},{"t":"h2","id":"the-metrics-computed","runs":[{"text":"The metrics, computed"}]},{"t":"p","runs":[{"text":"Accuracy","b":true},{"text":" = (TP + TN) / all = 978 / 1000 = "},{"text":"97.8%","b":true},{"text":". And the model that flags nothing scores 96%. Two points of improvement for all that work: this is the accuracy trap from the reuse lesson, in numbers."}]},{"t":"p","runs":[{"text":"Precision","b":true},{"text":" = TP / (TP + FP) = 24 / 30 = "},{"text":"80%","b":true},{"text":". Of what we flagged, 80% was fraud. This is the number the investigation team cares about, because it determines how much of their day is wasted."}]},{"t":"p","runs":[{"text":"Recall","b":true},{"text":" = TP / (TP + FN) = 24 / 40 = "},{"text":"60%","b":true},{"text":". Of all real fraud, we caught 60%. This is the number the finance team cares about, because 40% of the losses walked past."}]},{"t":"p","runs":[{"text":"F1","b":true},{"text":" = harmonic mean of precision and recall = 2 x (0.8 x 0.6) / (0.8 + 0.6) = "},{"text":"0.686","b":true},{"text":". The harmonic mean rather than the arithmetic one, because it punishes imbalance: a model with precision 1.0 and recall 0.02 has arithmetic mean 0.51 and F1 of 0.039. F1 refuses to be impressed by one number when the other is terrible."}]},{"t":"p","runs":[{"text":"Specificity","b":true},{"text":" = TN / (TN + FP) = 954 / 960 = "},{"text":"99.4%","b":true},{"text":". Of the innocent, how many we left alone. Standard in medicine, rarer in industry, and the complement of the false positive rate."}]},{"t":"h2","id":"the-two-directions-people-confuse","runs":[{"text":"The two directions people confuse"}]},{"t":"p","runs":[{"text":"Precision and recall are both about the positive class but they divide by different things, and mixing them up is the most common error in this area."}]},{"t":"ul","items":[[{"text":"Precision divides by "},{"text":"what the model said","b":true},{"text":". Column of the matrix."}],[{"text":"Recall divides by "},{"text":"what was true","b":true},{"text":". Row of the matrix."}]]},{"t":"p","runs":[{"text":"A sentence that fixes it permanently: "},{"text":"precision is how much of my alarm was justified; recall is how much of the danger I noticed.","i":true}]},{"t":"h2","id":"where-each-one-matters","runs":[{"text":"Where each one matters"}]},{"t":"p","runs":[{"text":"Recall matters most","b":true},{"text":" when a miss is catastrophic and a false alarm is merely annoying. Cancer screening: a missed tumour is fatal, a false positive is an unpleasant follow-up scan. Safety alerts. Content moderation for illegal material."}]},{"t":"p","runs":[{"text":"Precision matters most","b":true},{"text":" when acting on a positive is expensive or harmful. Sending a debt collector. Freezing an account. Any automated action against a person. If precision is 20%, four out of every five people you accuse are innocent, and that is a decision about people, not a metric."}]},{"t":"p","runs":[{"text":"Both, at a stated operating point","b":true},{"text":", is the honest report. “At a threshold of 0.62 we get 71% precision and 55% recall” tells someone what will actually happen. “F1 of 0.62” does not, and it hides which side of the trade-off you chose."}]},{"t":"h2","id":"the-number-that-is-not-on-this-list","runs":[{"text":"The number that is not on this list"}]},{"t":"p","runs":[{"text":"Most people asked “the model flagged this transaction: what is the chance it is really fraud?” reach for recall or the model’s accuracy. Neither answers it. The answer is precision, and precision depends on the base rate in a way that catches almost everyone."}]},{"t":"p","runs":[{"text":"Run the same model, 60% recall, 99.4% specificity, on a population where fraud is 0.1% rather than 4%. Out of 100,000 transactions, 100 are fraud. The model catches 60. It also flags 0.6% of the 99,900 clean ones, which is 599 false positives. Precision is 60 / 659 = "},{"text":"9%","b":true},{"text":"."}]},{"t":"p","runs":[{"text":"Same model, same sensitivity, same specificity. Precision fell from 80% to 9% because the base rate fell. This is the base rate fallacy, it is why a rare-disease screening test with excellent sensitivity still produces mostly false positives, and it is the single most important thing on this page."}]},{"t":"figure","label":"Figure","figure":{"cols":["Flagged","Not flagged"],"rows":["Actually fraud","Actually fine"],"type":"matrix","cells":[["60","40"],["599","99,301"]],"title":"The same model where fraud is 0.1 per cent of 100,000 transactions","caption":"Recall is still 60 per cent and specificity still 99.4 per cent. Precision is now 60 / 659 = 9 per cent, because 0.6 per cent of 99,900 clean transactions is 599 false alarms. Precision belongs to the population the model runs on, not to the model.","cellLabels":[["caught","missed"],["false alarms","left alone"]]},"alt":"Figure: The same model where fraud is 0.1 per cent of 100,000 transactions. A two by two grid, rows Actually fraud and Actually fine, columns Flagged and Not flagged: Actually fraud and Flagged: 60; Actually fraud and Not flagged: 40; Actually fine and Flagged: 599; Actually fine and Not flagged: 99,301. Recall is still 60 per cent and specificity still 99.4 per cent. Precision is now 60 / 659 = 9 per cent, because 0.6 per cent of 99,900 clean transactions is 599 false alarms. Precision belongs to the population the model runs on, not to the model."},{"t":"p","runs":[{"text":"So whenever someone quotes a precision figure, ask what the prevalence was in the set it was measured on. If your production prevalence differs, the precision you were promised does not transfer."}]},{"t":"h2","id":"getting-it-out-of-the-library","runs":[{"text":"Getting it out of the library"}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.metrics import confusion_matrix, classification_report\ntn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()\nprint(classification_report(y_true, y_pred, digits=3))"},{"t":"p","runs":[{"text":"Note the order that "},{"text":"ravel()","code":true},{"text":" gives you ("},{"text":"tn, fp, fn, tp","code":true},{"text":") which is not the order most people expect and is worth checking the first few times rather than assuming."}]}],"check":{"question":[{"text":"A screening model with 90% sensitivity and 95% specificity is moved from a high-risk clinic (10% prevalence) to general population screening (0.5% prevalence). What happens to the chance that a positive result is a true case?"}],"options":[{"text":[{"text":"It falls sharply, from about 67% to about 8%, because the far larger healthy population generates many more false positives even at an unchanged 5% false positive rate."}],"correct":true,"tempting":null},{"text":[{"text":"It stays the same, since sensitivity and specificity are properties of the test and do not change with population."}],"correct":false,"tempting":[{"text":"Correctly notes that sensitivity and specificity are population-independent, but the probability a positive is correct is not: it depends on prevalence."}]},{"text":[{"text":"It rises, because a lower prevalence means fewer diseased people to miss, so the positives that do appear are more reliable."}],"correct":false,"tempting":[{"text":"Reverses the arithmetic: fewer true cases means the same false positive rate produces a larger share of the positive results."}]},{"text":[{"text":"It cannot be determined without knowing the model’s accuracy on the new population."}],"correct":false,"tempting":[{"text":"Treats the calculation as under-determined when sensitivity, specificity and prevalence are sufficient to compute it exactly."}]}],"order":[3,1,2,0],"why":null},"prev":{"slug":"when-accuracy-lies","title":"When accuracy lies","minutes":9},"next":{"slug":"choosing-the-threshold","title":"The threshold is a business decision, not a default","minutes":9}}}