Predicting something rare, and why accuracy lies

The question, made specific

“Which customers are about to stop buying?” is the most commonly attempted prediction in small business, and the most commonly misread.

First it has to be made answerable. Stop buying is not a thing customers announce. You have to define it: a customer who ordered at least three times and has not ordered in 120 days, where 120 comes from your own data: look at the distribution of gaps between orders and pick a point well beyond where most repeat customers sit.

That definition is now the thing you are predicting. Change it and every number changes with it, which is why it goes in writing before anything is built.

The base rate decides how to read every figure

Suppose 5% of your customers lapse in any quarter.

A model that predicts “nobody will lapse”, always, is 95% accurate. It is also completely worthless. This is the accuracy paradox, and it is why accuracy is close to useless as a measure of anything rare.

Two figures replace it, and both are ordinary questions in plain words.

Of the customers it flagged, how many actually lapsed? That is precision, and it is the one that decides whether acting on the list is worth your time. Precision of 30% means two out of three people you contact were not going anywhere.

Of the customers who actually lapsed, how many did it flag? That is recall, and it decides how much of the problem you are catching. Recall of 40% means three out of five walked away unflagged.

You cannot have both. Flag more people and recall rises while precision falls; flag fewer and the reverse. There is no setting that escapes this, and a vendor quoting one figure without the other is quoting the flattering half.

Setting the threshold from what the action costs

A model outputs a score, not a decision. You choose the cut-off, and you choose it from arithmetic about your own business rather than from a default.

Say a flagged customer gets a phone call and a 10% voucher. The call costs you ten minutes; the voucher costs margin only if they were coming back anyway. A retained customer is worth USD 400 over the next year.

Then a wrong flag is cheap and a missed lapse is expensive, so you set the threshold low, accept precision around 25%, and catch more of them.

Reverse it (the intervention is an expensive hamper and a visit), and a wrong flag hurts, so you raise the threshold, contact fewer people, and accept that you will miss most lapses.

This is the whole of threshold-setting, and it is a business decision wearing a technical costume. Nobody but you can make it.

Before modelling: the rule that usually wins

Sort customers by days since last order, descending. Take the ones past your threshold who used to order regularly. That is your list.

For most small businesses this simple rule captures the large majority of what a model would find, costs nothing, and is explicable to whoever makes the calls. Build it first and measure it. Then, if you want, see whether a model beats it: using a holdout period, as in the previous lesson, because a model scored on the customers it learned from will look magnificent.

What actually moves the number

Not the algorithm. The features: the columns you give it.

Recency, frequency and monetary value, computed per customer, carry most of the signal in most businesses. Add gap-to-previous-gap: a customer whose intervals are lengthening is a useful signal and almost nobody computes it. Add whether their last experience involved a complaint, a delay or a substitution.

An hour spent building those columns is worth more than any amount of model selection, and this is the general rule in applied work: the data beats the method, nearly always.

The uncomfortable part

A prediction about a person is a decision about how to treat them, and that deserves a moment’s thought.

Do not use a model like this to decide who gets worse service, slower responses, or a higher price. Beyond the fairness of it, which matters, there is a mechanical problem: the model learned from your past behaviour, so if you historically gave less attention to a group, the model will predict they were leaving and recommend you stop trying, and the prediction will come true because you acted on it.

Use it to decide who gets extra attention. That way a wrong prediction costs you a phone call and gives somebody an unexpectedly good experience, which is the cheapest kind of error to make.