{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"choosing-the-threshold","title":"The threshold is a business decision, not a default","url":"https://addaly.com/learn/machine-learning-foundations/choosing-the-threshold","position":44,"of":92,"minutes":9,"answer":"The model produces a score and the threshold converts it into a decision; choose the threshold from the relative cost of the two errors, the capacity to act, or a stated requirement.","keyIdea":"The model produces a score and the threshold converts it into a decision; choose the threshold from the relative cost of the two errors, the capacity to act, or a stated requirement, never from the library default."},"body":{"format":"markdown","text":"## Where 0.5 came from\n\nNowhere useful. `predict()` applies 0.5 because a library needs a default, not because 0.5 is right for your problem. It is right only in the narrow case where the two error types cost the same and the classes are balanced, which describes almost nothing.\n\nEverything else in this lesson follows from separating two things that beginners treat as one: **the model produces a score; the threshold turns it into a decision.** They are tuned separately, by different people, for different reasons, and the threshold can be changed at any time without retraining.\n\n## Moving it, and watching what happens\n\nTake the fraud model from the last lesson and sweep the threshold:\n\n```python\nimport numpy as np\nfrom sklearn.metrics import precision_score, recall_score\np = model.predict_proba(X_val)[:, 1]\nfor t in np.arange(0.1, 0.95, 0.05):\n    pred = (p >= t).astype(int)\n    print(f\"{t:.2f}  precision {precision_score(y_val, pred):.3f}\"\n          f\"  recall {recall_score(y_val, pred):.3f}\")\n```\n\nThe pattern is always the same. **Lower threshold: more flags, higher recall, lower precision.** **Higher threshold: fewer flags, higher precision, lower recall.** There is no setting that improves both; that would require a better model.\n\nA typical run might give you: at 0.20, precision 0.41 and recall 0.88. At 0.50, precision 0.80 and recall 0.60. At 0.75, precision 0.94 and recall 0.31. Those are three different products.\n\n## Choosing by expected cost\n\nThe principled method. Assign a cost to each error type and pick the threshold that minimises total expected cost.\n\nSuppose a missed fraud costs ₹18,000 on average, and a false alarm costs ₹90 of review time. Then:\n\n```python\ncost = 18000 * fn_count + 90 * fp_count\n```\n\nEvaluate that across the threshold sweep and take the minimum. Because a miss costs 200 times a false alarm, the optimal threshold lands far below 0.5, probably nearer 0.05, and the model will flag many more transactions than it did at the default.\n\n> Figure: Total cost of mistakes at four thresholds. Bars (₹ thousands of expected cost per 1,000 transactions): 0.05: 221 false alarms, 1 miss 38; 0.20: 50 false alarms, 5 misses 95; 0.50: 6 false alarms, 16 misses 289; 0.75: 1 false alarm, 28 misses 504. A miss costs ₹18,000 and a false alarm ₹90, so the minimum sits far below the library default and the model flags many more transactions than it did at 0.5. Change the 200-to-1 ratio and the winner moves, which is the conversation worth having with whoever owns those costs.\n\nFor a well-calibrated model there is a closed form. The optimal threshold is\n\n```text\nt = cost(FP) / (cost(FP) + cost(FN))\n```\n\nWith 90 and 18,000 that gives `90 / 18090 = 0.005`. Note the condition: *well-calibrated*. If your probabilities do not mean what they say, this formula gives you a threshold on a scale that does not exist, which is the subject of a later lesson in this module.\n\n## Choosing by capacity\n\nOften the constraint is not cost but throughput. The fraud team can review 200 cases a day. Then the threshold is whatever produces 200 flags, and the metric is precision at 200: how many of that day's investigations were worthwhile.\n\nThis reframing is usually closer to how the organisation actually works, and it changes what you optimise. You no longer care about the model's behaviour across the whole score range, only about the ranking at the top. A model that is better at the very top and worse elsewhere is the better model here, and average metrics will not tell you that.\n\n## Choosing by a fixed requirement\n\nSometimes one side is dictated. \"We must catch at least 95% of cases\" from a regulator, or \"no more than 2% of legitimate users may be blocked\" from a product owner. Then set the threshold to satisfy the constraint on validation data and report what the other side costs you.\n\n```python\nfrom sklearn.metrics import precision_recall_curve\nprec, rec, thr = precision_recall_curve(y_val, p)\ni = np.argmax(rec >= 0.95)     # first threshold meeting the requirement\nprint(thr[i], prec[i])\n```\n\n## Three failures worth avoiding\n\n**Choosing the threshold on the test set.** It is a fitted parameter like any other. Choose it on validation data, then measure the chosen threshold's performance on test.\n\n**Assuming it transfers.** A threshold tuned on last quarter's data assumes this quarter has the same score distribution. When the population shifts, the same threshold produces a different flag rate: often the first symptom of drift anyone notices, and a good thing to monitor.\n\n**Reporting only the number at 0.5.** If you report \"precision 0.80, recall 0.60\" without saying which threshold produced it, you have reported one arbitrary point on a curve and hidden the rest.\n\n## The output that is not a decision\n\nA final option that is often the best one: do not threshold at all. Show the score.\n\nFor a triage queue, a ranked list is more useful than a binary flag, because the human at the other end applies judgement that the threshold would have thrown away. For a risk score shown to an underwriter, a band (low, medium, high) carries more than a yes or no. Forcing a continuous output into two buckets destroys information, and the only reason to do it is that some downstream system requires a decision. When no such system exists, do not invent one."},"check":{"question":"A team wants to minimise expected cost where a false negative costs 40 times a false positive. Their model's probabilities are well calibrated. Roughly where should the threshold sit and why?","options":[{"text":"At 0.5, since a calibrated model is already optimal at the default threshold.","correct":false,"tempting":"Calibration makes the probabilities meaningful but says nothing about where to cut them; 0.5 is optimal only when the two errors cost the same."},{"text":"Around 0.025, because the optimal threshold is cost(FP) divided by the sum of both costs, so a much cheaper false positive pushes the cut far down.","correct":true},{"text":"Around 0.975, because the expensive error should require a high level of confidence before the model commits.","correct":false,"tempting":"Inverts the direction: raising the threshold produces fewer positive predictions and therefore more of the expensive false negatives."},{"text":"Around 0.4, since the threshold should move modestly from the default in proportion to the log of the cost ratio.","correct":false,"tempting":"Invents a logarithmic adjustment; the relationship is a simple ratio and produces a far more extreme threshold than a modest shift."}]},"prev":{"slug":"confusion-matrix-worked","title":"The four numbers, worked through","url":"https://addaly.com/learn/machine-learning-foundations/confusion-matrix-worked"},"next":{"slug":"roc-and-pr-curves","title":"ROC and precision-recall: every threshold at once","url":"https://addaly.com/learn/machine-learning-foundations/roc-and-pr-curves"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The threshold is a business decision, not a default', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/choosing-the-threshold (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The threshold is a business decision, not a default","courseTitle":"Machine Learning, Foundations","module":{"index":5,"slug":"measuring-honestly","title":"Measuring a model honestly"},"answer":null,"keyIdea":"The model produces a score and the threshold converts it into a decision; choose the threshold from the relative cost of the two errors, the capacity to act, or a stated requirement, never from the library default.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"₹ thousands of expected cost per 1,000 transactions","title":"Total cost of mistakes at four thresholds","series":[{"label":"0.05: 221 false alarms, 1 miss","value":38,"highlight":true},{"label":"0.20: 50 false alarms, 5 misses","value":95},{"label":"0.50: 6 false alarms, 16 misses","value":289},{"label":"0.75: 1 false alarm, 28 misses","value":504}],"caption":"A miss costs ₹18,000 and a false alarm ₹90, so the minimum sits far below the library default and the model flags many more transactions than it did at 0.5. Change the 200-to-1 ratio and the winner moves, which is the conversation worth having with whoever owns those costs."},"alt":"Figure: Total cost of mistakes at four thresholds. Bars (₹ thousands of expected cost per 1,000 transactions): 0.05: 221 false alarms, 1 miss 38; 0.20: 50 false alarms, 5 misses 95; 0.50: 6 false alarms, 16 misses 289; 0.75: 1 false alarm, 28 misses 504. A miss costs ₹18,000 and a false alarm ₹90, so the minimum sits far below the library default and the model flags many more transactions than it did at 0.5. Change the 200-to-1 ratio and the winner moves, which is the conversation worth having with whoever owns those costs."}},"blocks":[{"t":"h2","id":"where-0-5-came-from","runs":[{"text":"Where 0.5 came from"}]},{"t":"p","runs":[{"text":"Nowhere useful. "},{"text":"predict()","code":true},{"text":" applies 0.5 because a library needs a default, not because 0.5 is right for your problem. It is right only in the narrow case where the two error types cost the same and the classes are balanced, which describes almost nothing."}]},{"t":"p","runs":[{"text":"Everything else in this lesson follows from separating two things that beginners treat as one: "},{"text":"the model produces a score; the threshold turns it into a decision.","b":true},{"text":" They are tuned separately, by different people, for different reasons, and the threshold can be changed at any time without retraining."}]},{"t":"h2","id":"moving-it-and-watching-what-happens","runs":[{"text":"Moving it, and watching what happens"}]},{"t":"p","runs":[{"text":"Take the fraud model from the last lesson and sweep the threshold:"}]},{"t":"code","lang":"python","label":"Python","text":"import numpy as np\nfrom sklearn.metrics import precision_score, recall_score\np = model.predict_proba(X_val)[:, 1]\nfor t in np.arange(0.1, 0.95, 0.05):\n    pred = (p >= t).astype(int)\n    print(f\"{t:.2f}  precision {precision_score(y_val, pred):.3f}\"\n          f\"  recall {recall_score(y_val, pred):.3f}\")"},{"t":"p","runs":[{"text":"The pattern is always the same. "},{"text":"Lower threshold: more flags, higher recall, lower precision.","b":true},{"text":" "},{"text":"Higher threshold: fewer flags, higher precision, lower recall.","b":true},{"text":" There is no setting that improves both; that would require a better model."}]},{"t":"p","runs":[{"text":"A typical run might give you: at 0.20, precision 0.41 and recall 0.88. At 0.50, precision 0.80 and recall 0.60. At 0.75, precision 0.94 and recall 0.31. Those are three different products."}]},{"t":"h2","id":"choosing-by-expected-cost","runs":[{"text":"Choosing by expected cost"}]},{"t":"p","runs":[{"text":"The principled method. Assign a cost to each error type and pick the threshold that minimises total expected cost."}]},{"t":"p","runs":[{"text":"Suppose a missed fraud costs ₹18,000 on average, and a false alarm costs ₹90 of review time. Then:"}]},{"t":"code","lang":"python","label":"Python","text":"cost = 18000 * fn_count + 90 * fp_count"},{"t":"p","runs":[{"text":"Evaluate that across the threshold sweep and take the minimum. Because a miss costs 200 times a false alarm, the optimal threshold lands far below 0.5, probably nearer 0.05, and the model will flag many more transactions than it did at the default."}]},{"t":"p","runs":[{"text":"For a well-calibrated model there is a closed form. The optimal threshold is"}]},{"t":"code","lang":"text","label":"Text","text":"t = cost(FP) / (cost(FP) + cost(FN))"},{"t":"p","runs":[{"text":"With 90 and 18,000 that gives "},{"text":"90 / 18090 = 0.005","code":true},{"text":". Note the condition: "},{"text":"well-calibrated","i":true},{"text":". If your probabilities do not mean what they say, this formula gives you a threshold on a scale that does not exist, which is the subject of a later lesson in this module."}]},{"t":"h2","id":"choosing-by-capacity","runs":[{"text":"Choosing by capacity"}]},{"t":"p","runs":[{"text":"Often the constraint is not cost but throughput. The fraud team can review 200 cases a day. Then the threshold is whatever produces 200 flags, and the metric is precision at 200: how many of that day’s investigations were worthwhile."}]},{"t":"p","runs":[{"text":"This reframing is usually closer to how the organisation actually works, and it changes what you optimise. You no longer care about the model’s behaviour across the whole score range, only about the ranking at the top. A model that is better at the very top and worse elsewhere is the better model here, and average metrics will not tell you that."}]},{"t":"h2","id":"choosing-by-a-fixed-requirement","runs":[{"text":"Choosing by a fixed requirement"}]},{"t":"p","runs":[{"text":"Sometimes one side is dictated. “We must catch at least 95% of cases” from a regulator, or “no more than 2% of legitimate users may be blocked” from a product owner. Then set the threshold to satisfy the constraint on validation data and report what the other side costs you."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.metrics import precision_recall_curve\nprec, rec, thr = precision_recall_curve(y_val, p)\ni = np.argmax(rec >= 0.95)     # first threshold meeting the requirement\nprint(thr[i], prec[i])"},{"t":"h2","id":"three-failures-worth-avoiding","runs":[{"text":"Three failures worth avoiding"}]},{"t":"p","runs":[{"text":"Choosing the threshold on the test set.","b":true},{"text":" It is a fitted parameter like any other. Choose it on validation data, then measure the chosen threshold’s performance on test."}]},{"t":"p","runs":[{"text":"Assuming it transfers.","b":true},{"text":" A threshold tuned on last quarter’s data assumes this quarter has the same score distribution. When the population shifts, the same threshold produces a different flag rate: often the first symptom of drift anyone notices, and a good thing to monitor."}]},{"t":"p","runs":[{"text":"Reporting only the number at 0.5.","b":true},{"text":" If you report “precision 0.80, recall 0.60” without saying which threshold produced it, you have reported one arbitrary point on a curve and hidden the rest."}]},{"t":"h2","id":"the-output-that-is-not-a-decision","runs":[{"text":"The output that is not a decision"}]},{"t":"p","runs":[{"text":"A final option that is often the best one: do not threshold at all. Show the score."}]},{"t":"p","runs":[{"text":"For a triage queue, a ranked list is more useful than a binary flag, because the human at the other end applies judgement that the threshold would have thrown away. For a risk score shown to an underwriter, a band (low, medium, high) carries more than a yes or no. Forcing a continuous output into two buckets destroys information, and the only reason to do it is that some downstream system requires a decision. When no such system exists, do not invent one."}]}],"check":{"question":[{"text":"A team wants to minimise expected cost where a false negative costs 40 times a false positive. Their model’s probabilities are well calibrated. Roughly where should the threshold sit and why?"}],"options":[{"text":[{"text":"At 0.5, since a calibrated model is already optimal at the default threshold."}],"correct":false,"tempting":[{"text":"Calibration makes the probabilities meaningful but says nothing about where to cut them; 0.5 is optimal only when the two errors cost the same."}]},{"text":[{"text":"Around 0.025, because the optimal threshold is cost(FP) divided by the sum of both costs, so a much cheaper false positive pushes the cut far down."}],"correct":true,"tempting":null},{"text":[{"text":"Around 0.975, because the expensive error should require a high level of confidence before the model commits."}],"correct":false,"tempting":[{"text":"Inverts the direction: raising the threshold produces fewer positive predictions and therefore more of the expensive false negatives."}]},{"text":[{"text":"Around 0.4, since the threshold should move modestly from the default in proportion to the log of the cost ratio."}],"correct":false,"tempting":[{"text":"Invents a logarithmic adjustment; the relationship is a simple ratio and produces a far more extreme threshold than a modest shift."}]}],"order":[0,1,2,3],"why":null},"prev":{"slug":"confusion-matrix-worked","title":"The four numbers, worked through","minutes":9},"next":{"slug":"roc-and-pr-curves","title":"ROC and precision-recall: every threshold at once","minutes":9}}}