{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"cost-sensitive-decisions","title":"From a probability to a decision worth making","url":"https://addaly.com/learn/machine-learning-foundations/cost-sensitive-decisions","position":49,"of":92,"minutes":8,"answer":"The break-even threshold is the cost of a false positive divided by the sum of both error costs, it can and usually should be computed per row from that row's stakes.","keyIdea":"The break-even threshold is the cost of a false positive divided by the sum of both error costs, it can and usually should be computed per row from that row's stakes, and the arithmetic is only valid for whoever's costs were put into it."},"body":{"format":"markdown","text":"## The last step nobody teaches\n\nA model outputs 0.34. What do you do?\n\nEvery lesson so far has been about producing and checking that number. This one is about the arithmetic that turns it into an action, and it is short because the arithmetic is simple. What is not simple is getting the costs.\n\n## Expected value\n\nFor each available action, compute the expected outcome under the model's probability, and take the best one.\n\nA lender considers approving a ₹200,000 loan. If repaid, profit is ₹24,000. If defaulted, loss is ₹150,000. The model says the default probability is 0.09.\n\n```text\nExpected value of approving = 0.91 * 24,000 - 0.09 * 150,000\n                            = 21,840 - 13,500 = +8,340\nExpected value of declining = 0\n```\n\nApprove. And the break-even default probability is where the two are equal:\n\n```text\np * 150,000 = (1 - p) * 24,000\np = 24,000 / 174,000 = 0.138\n```\n\nApprove below 13.8%, decline above. That is the threshold, derived from the business rather than chosen by a library, and it is defensible in a room full of people who do not know what AUC is.\n\n## Costs are usually per-row, not global\n\nThe fraud example in the threshold lesson used one average cost per miss. Real costs vary by row: missing a ₹500 fraud and a ₹5,00,000 fraud are not the same event.\n\nSo compute the threshold per row:\n\n```python\nthreshold = review_cost / (review_cost + transaction_amount)\nflag = p_fraud > threshold\n```\n\nA ₹90 review against a ₹5,00,000 transaction gives a threshold of 0.00018: flag on almost any suspicion. Against a ₹500 transaction it gives 0.15: only flag if reasonably confident. One line of code, and it typically outperforms a globally tuned threshold by a wide margin, because it is using information the global threshold throws away.\n\nThis is the most under-used idea in applied classification. It also requires the probabilities to be calibrated, which is why that lesson came first.\n\n## More than two actions\n\nMost real systems have three or more: approve, decline, and refer to a human. Then you have two thresholds, and the middle band is where the model defers.\n\nThe width of that band is set by review capacity and by the value of a human decision. If a reviewer is right 95% of the time and costs ₹200, you can compute how wide the band should be, and it turns out to depend on how much the model's uncertainty in that region actually costs you.\n\nThis \"reject option\" framing (a classifier allowed to abstain) is standard in medical decision support and in credit, and it is usually a better design than forcing a binary decision. The metric that goes with it is the **coverage-accuracy curve**: accuracy on the cases the model chose to decide, plotted against the fraction it decided. A model that is 99% accurate on the 60% of cases it is confident about, and defers the rest, is a useful product even if its overall accuracy is unremarkable.\n\n## Where the costs come from\n\nThe hard part is not the arithmetic. It is that nobody has written the costs down.\n\nAsk anyway, and ask badly rather than not at all. \"If we wrongly decline a good customer, roughly what does that cost us?\" produces an argument, and the argument is the useful output: it surfaces that the marketing team values a customer's lifetime and the risk team values this quarter's losses, and that they have never had to reconcile those numbers before.\n\nWhen a cost is unknowable, use a ratio instead of an absolute. \"A missed fraud is roughly 200 times a false alarm\" is easier to agree on than two rupee figures, and it is all the threshold formula needs.\n\n## The costs that do not go in the spreadsheet\n\nSome consequences are not priced, and pretending otherwise is how models cause harm.\n\nA wrongly declined loan may cost the lender ₹24,000 of foregone profit. It may cost the applicant a business that does not open. Those two numbers are not on the same scale and cannot be added.\n\nWhen the cost falls on a person outside the organisation, expected-value arithmetic optimises the organisation's outcome and is silent about theirs. This is not an argument against doing the arithmetic; it is an argument for stating whose costs are in the sum and whose are not, and for treating some errors as constraints rather than terms. Some things belong in the model as a hard limit (\"never decline without a stated reason\", \"never act automatically above this amount\") precisely because they should not be traded off against anything."},"check":{"question":"A retailer flags orders for manual review at a fixed probability threshold. Someone proposes computing the threshold per order as review_cost / (review_cost + order_value). Why does this usually outperform the fixed threshold?","options":[{"text":"It removes the need for calibrated probabilities, since the threshold now adapts to each order.","correct":false,"tempting":"Inverts the dependency: a per-row threshold relies more heavily on calibrated probabilities, not less, because the cut point is now computed rather than tuned."},{"text":"It uses each order's own stakes, so cheap orders need strong evidence before a review is worth it while expensive ones justify a review on faint suspicion: information a single global cut discards.","correct":true},{"text":"It increases the total number of reviews, which raises recall across the board.","correct":false,"tempting":"The rule raises the flag rate for large orders and lowers it for small ones; the total can move either way and higher volume is not the source of the gain."},{"text":"It makes the classifier cost-sensitive during training, so the model itself learns to weight expensive orders more.","correct":false,"tempting":"Confuses a decision rule applied after prediction with a change to the training objective; the model is untouched by this."}]},"prev":{"slug":"comparing-two-models","title":"Is that improvement real?","url":"https://addaly.com/learn/machine-learning-foundations/comparing-two-models"},"next":{"slug":"fairness-you-can-compute","title":"Fairness as arithmetic: the definitions and what each one demands","url":"https://addaly.com/learn/machine-learning-foundations/fairness-you-can-compute"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'From a probability to a decision worth making', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/cost-sensitive-decisions (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"From a probability to a decision worth making","courseTitle":"Machine Learning, Foundations","module":{"index":5,"slug":"measuring-honestly","title":"Measuring a model honestly"},"answer":null,"keyIdea":"The break-even threshold is the cost of a false positive divided by the sum of both error costs, it can and usually should be computed per row from that row’s stakes, and the arithmetic is only valid for whoever’s costs were put into it.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-last-step-nobody-teaches","runs":[{"text":"The last step nobody teaches"}]},{"t":"p","runs":[{"text":"A model outputs 0.34. What do you do?"}]},{"t":"p","runs":[{"text":"Every lesson so far has been about producing and checking that number. This one is about the arithmetic that turns it into an action, and it is short because the arithmetic is simple. What is not simple is getting the costs."}]},{"t":"h2","id":"expected-value","runs":[{"text":"Expected value"}]},{"t":"p","runs":[{"text":"For each available action, compute the expected outcome under the model’s probability, and take the best one."}]},{"t":"p","runs":[{"text":"A lender considers approving a ₹200,000 loan. If repaid, profit is ₹24,000. If defaulted, loss is ₹150,000. The model says the default probability is 0.09."}]},{"t":"code","lang":"text","label":"Text","text":"Expected value of approving = 0.91 * 24,000 - 0.09 * 150,000\n                            = 21,840 - 13,500 = +8,340\nExpected value of declining = 0"},{"t":"p","runs":[{"text":"Approve. And the break-even default probability is where the two are equal:"}]},{"t":"code","lang":"text","label":"Text","text":"p * 150,000 = (1 - p) * 24,000\np = 24,000 / 174,000 = 0.138"},{"t":"p","runs":[{"text":"Approve below 13.8%, decline above. That is the threshold, derived from the business rather than chosen by a library, and it is defensible in a room full of people who do not know what AUC is."}]},{"t":"h2","id":"costs-are-usually-per-row-not-global","runs":[{"text":"Costs are usually per-row, not global"}]},{"t":"p","runs":[{"text":"The fraud example in the threshold lesson used one average cost per miss. Real costs vary by row: missing a ₹500 fraud and a ₹5,00,000 fraud are not the same event."}]},{"t":"p","runs":[{"text":"So compute the threshold per row:"}]},{"t":"code","lang":"python","label":"Python","text":"threshold = review_cost / (review_cost + transaction_amount)\nflag = p_fraud > threshold"},{"t":"p","runs":[{"text":"A ₹90 review against a ₹5,00,000 transaction gives a threshold of 0.00018: flag on almost any suspicion. Against a ₹500 transaction it gives 0.15: only flag if reasonably confident. One line of code, and it typically outperforms a globally tuned threshold by a wide margin, because it is using information the global threshold throws away."}]},{"t":"p","runs":[{"text":"This is the most under-used idea in applied classification. It also requires the probabilities to be calibrated, which is why that lesson came first."}]},{"t":"h2","id":"more-than-two-actions","runs":[{"text":"More than two actions"}]},{"t":"p","runs":[{"text":"Most real systems have three or more: approve, decline, and refer to a human. Then you have two thresholds, and the middle band is where the model defers."}]},{"t":"p","runs":[{"text":"The width of that band is set by review capacity and by the value of a human decision. If a reviewer is right 95% of the time and costs ₹200, you can compute how wide the band should be, and it turns out to depend on how much the model’s uncertainty in that region actually costs you."}]},{"t":"p","runs":[{"text":"This “reject option” framing (a classifier allowed to abstain) is standard in medical decision support and in credit, and it is usually a better design than forcing a binary decision. The metric that goes with it is the "},{"text":"coverage-accuracy curve","b":true},{"text":": accuracy on the cases the model chose to decide, plotted against the fraction it decided. A model that is 99% accurate on the 60% of cases it is confident about, and defers the rest, is a useful product even if its overall accuracy is unremarkable."}]},{"t":"h2","id":"where-the-costs-come-from","runs":[{"text":"Where the costs come from"}]},{"t":"p","runs":[{"text":"The hard part is not the arithmetic. It is that nobody has written the costs down."}]},{"t":"p","runs":[{"text":"Ask anyway, and ask badly rather than not at all. “If we wrongly decline a good customer, roughly what does that cost us?” produces an argument, and the argument is the useful output: it surfaces that the marketing team values a customer’s lifetime and the risk team values this quarter’s losses, and that they have never had to reconcile those numbers before."}]},{"t":"p","runs":[{"text":"When a cost is unknowable, use a ratio instead of an absolute. “A missed fraud is roughly 200 times a false alarm” is easier to agree on than two rupee figures, and it is all the threshold formula needs."}]},{"t":"h2","id":"the-costs-that-do-not-go-in-the-spreadsheet","runs":[{"text":"The costs that do not go in the spreadsheet"}]},{"t":"p","runs":[{"text":"Some consequences are not priced, and pretending otherwise is how models cause harm."}]},{"t":"p","runs":[{"text":"A wrongly declined loan may cost the lender ₹24,000 of foregone profit. It may cost the applicant a business that does not open. Those two numbers are not on the same scale and cannot be added."}]},{"t":"p","runs":[{"text":"When the cost falls on a person outside the organisation, expected-value arithmetic optimises the organisation’s outcome and is silent about theirs. This is not an argument against doing the arithmetic; it is an argument for stating whose costs are in the sum and whose are not, and for treating some errors as constraints rather than terms. Some things belong in the model as a hard limit (“never decline without a stated reason”, “never act automatically above this amount”) precisely because they should not be traded off against anything."}]}],"check":{"question":[{"text":"A retailer flags orders for manual review at a fixed probability threshold. Someone proposes computing the threshold per order as review_cost / (review_cost + order_value). Why does this usually outperform the fixed threshold?"}],"options":[{"text":[{"text":"It removes the need for calibrated probabilities, since the threshold now adapts to each order."}],"correct":false,"tempting":[{"text":"Inverts the dependency: a per-row threshold relies more heavily on calibrated probabilities, not less, because the cut point is now computed rather than tuned."}]},{"text":[{"text":"It uses each order’s own stakes, so cheap orders need strong evidence before a review is worth it while expensive ones justify a review on faint suspicion: information a single global cut discards."}],"correct":true,"tempting":null},{"text":[{"text":"It increases the total number of reviews, which raises recall across the board."}],"correct":false,"tempting":[{"text":"The rule raises the flag rate for large orders and lowers it for small ones; the total can move either way and higher volume is not the source of the gain."}]},{"text":[{"text":"It makes the classifier cost-sensitive during training, so the model itself learns to weight expensive orders more."}],"correct":false,"tempting":[{"text":"Confuses a decision rule applied after prediction with a change to the training objective; the model is untouched by this."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"comparing-two-models","title":"Is that improvement real?","minutes":9},"next":{"slug":"fairness-you-can-compute","title":"Fairness as arithmetic: the definitions and what each one demands","minutes":10}}}