From a probability to a decision worth making

The last step nobody teaches

A model outputs 0.34. What do you do?

Every lesson so far has been about producing and checking that number. This one is about the arithmetic that turns it into an action, and it is short because the arithmetic is simple. What is not simple is getting the costs.

Expected value

For each available action, compute the expected outcome under the model’s probability, and take the best one.

A lender considers approving a ₹200,000 loan. If repaid, profit is ₹24,000. If defaulted, loss is ₹150,000. The model says the default probability is 0.09.

Expected value of approving = 0.91 * 24,000 - 0.09 * 150,000
                            = 21,840 - 13,500 = +8,340
Expected value of declining = 0

Approve. And the break-even default probability is where the two are equal:

p * 150,000 = (1 - p) * 24,000
p = 24,000 / 174,000 = 0.138

Approve below 13.8%, decline above. That is the threshold, derived from the business rather than chosen by a library, and it is defensible in a room full of people who do not know what AUC is.

Costs are usually per-row, not global

The fraud example in the threshold lesson used one average cost per miss. Real costs vary by row: missing a ₹500 fraud and a ₹5,00,000 fraud are not the same event.

So compute the threshold per row:

threshold = review_cost / (review_cost + transaction_amount)
flag = p_fraud > threshold

A ₹90 review against a ₹5,00,000 transaction gives a threshold of 0.00018: flag on almost any suspicion. Against a ₹500 transaction it gives 0.15: only flag if reasonably confident. One line of code, and it typically outperforms a globally tuned threshold by a wide margin, because it is using information the global threshold throws away.

This is the most under-used idea in applied classification. It also requires the probabilities to be calibrated, which is why that lesson came first.

More than two actions

Most real systems have three or more: approve, decline, and refer to a human. Then you have two thresholds, and the middle band is where the model defers.

The width of that band is set by review capacity and by the value of a human decision. If a reviewer is right 95% of the time and costs ₹200, you can compute how wide the band should be, and it turns out to depend on how much the model’s uncertainty in that region actually costs you.

This “reject option” framing (a classifier allowed to abstain) is standard in medical decision support and in credit, and it is usually a better design than forcing a binary decision. The metric that goes with it is the coverage-accuracy curve: accuracy on the cases the model chose to decide, plotted against the fraction it decided. A model that is 99% accurate on the 60% of cases it is confident about, and defers the rest, is a useful product even if its overall accuracy is unremarkable.

Where the costs come from

The hard part is not the arithmetic. It is that nobody has written the costs down.

Ask anyway, and ask badly rather than not at all. “If we wrongly decline a good customer, roughly what does that cost us?” produces an argument, and the argument is the useful output: it surfaces that the marketing team values a customer’s lifetime and the risk team values this quarter’s losses, and that they have never had to reconcile those numbers before.

When a cost is unknowable, use a ratio instead of an absolute. “A missed fraud is roughly 200 times a false alarm” is easier to agree on than two rupee figures, and it is all the threshold formula needs.

The costs that do not go in the spreadsheet

Some consequences are not priced, and pretending otherwise is how models cause harm.

A wrongly declined loan may cost the lender ₹24,000 of foregone profit. It may cost the applicant a business that does not open. Those two numbers are not on the same scale and cannot be added.

When the cost falls on a person outside the organisation, expected-value arithmetic optimises the organisation’s outcome and is silent about theirs. This is not an argument against doing the arithmetic; it is an argument for stating whose costs are in the sum and whose are not, and for treating some errors as constraints rather than terms. Some things belong in the model as a hard limit (“never decline without a stated reason”, “never act automatically above this amount”) precisely because they should not be traded off against anything.