The threshold is a business decision, not a default

Where 0.5 came from

Nowhere useful. predict() applies 0.5 because a library needs a default, not because 0.5 is right for your problem. It is right only in the narrow case where the two error types cost the same and the classes are balanced, which describes almost nothing.

Everything else in this lesson follows from separating two things that beginners treat as one: the model produces a score; the threshold turns it into a decision. They are tuned separately, by different people, for different reasons, and the threshold can be changed at any time without retraining.

Moving it, and watching what happens

Take the fraud model from the last lesson and sweep the threshold:

import numpy as np
from sklearn.metrics import precision_score, recall_score
p = model.predict_proba(X_val)[:, 1]
for t in np.arange(0.1, 0.95, 0.05):
    pred = (p >= t).astype(int)
    print(f"{t:.2f}  precision {precision_score(y_val, pred):.3f}"
          f"  recall {recall_score(y_val, pred):.3f}")

The pattern is always the same. Lower threshold: more flags, higher recall, lower precision. Higher threshold: fewer flags, higher precision, lower recall. There is no setting that improves both; that would require a better model.

A typical run might give you: at 0.20, precision 0.41 and recall 0.88. At 0.50, precision 0.80 and recall 0.60. At 0.75, precision 0.94 and recall 0.31. Those are three different products.

Choosing by expected cost

The principled method. Assign a cost to each error type and pick the threshold that minimises total expected cost.

Suppose a missed fraud costs ₹18,000 on average, and a false alarm costs ₹90 of review time. Then:

cost = 18000 * fn_count + 90 * fp_count

Evaluate that across the threshold sweep and take the minimum. Because a miss costs 200 times a false alarm, the optimal threshold lands far below 0.5, probably nearer 0.05, and the model will flag many more transactions than it did at the default.

For a well-calibrated model there is a closed form. The optimal threshold is

t = cost(FP) / (cost(FP) + cost(FN))

With 90 and 18,000 that gives 90 / 18090 = 0.005. Note the condition: well-calibrated. If your probabilities do not mean what they say, this formula gives you a threshold on a scale that does not exist, which is the subject of a later lesson in this module.

Choosing by capacity

Often the constraint is not cost but throughput. The fraud team can review 200 cases a day. Then the threshold is whatever produces 200 flags, and the metric is precision at 200: how many of that day’s investigations were worthwhile.

This reframing is usually closer to how the organisation actually works, and it changes what you optimise. You no longer care about the model’s behaviour across the whole score range, only about the ranking at the top. A model that is better at the very top and worse elsewhere is the better model here, and average metrics will not tell you that.

Choosing by a fixed requirement

Sometimes one side is dictated. “We must catch at least 95% of cases” from a regulator, or “no more than 2% of legitimate users may be blocked” from a product owner. Then set the threshold to satisfy the constraint on validation data and report what the other side costs you.

from sklearn.metrics import precision_recall_curve
prec, rec, thr = precision_recall_curve(y_val, p)
i = np.argmax(rec >= 0.95)     # first threshold meeting the requirement
print(thr[i], prec[i])

Three failures worth avoiding

Choosing the threshold on the test set. It is a fitted parameter like any other. Choose it on validation data, then measure the chosen threshold’s performance on test.

Assuming it transfers. A threshold tuned on last quarter’s data assumes this quarter has the same score distribution. When the population shifts, the same threshold produces a different flag rate: often the first symptom of drift anyone notices, and a good thing to monitor.

Reporting only the number at 0.5. If you report “precision 0.80, recall 0.60” without saying which threshold produced it, you have reported one arbitrary point on a curve and hidden the rest.

The output that is not a decision

A final option that is often the best one: do not threshold at all. Show the score.

For a triage queue, a ranked list is more useful than a binary flag, because the human at the other end applies judgement that the threshold would have thrown away. For a risk score shown to an underwriter, a band (low, medium, high) carries more than a yes or no. Forcing a continuous output into two buckets destroys information, and the only reason to do it is that some downstream system requires a decision. When no such system exists, do not invent one.