{"course":{"slug":"ai-for-small-business","title":"AI for a Small Business","url":"https://addaly.com/learn/ai-for-small-business","level":"beginner","track":"work"},"lesson":{"slug":"predicting-who-leaves","title":"Predicting something rare, and why accuracy lies","url":"https://addaly.com/learn/ai-for-small-business/predicting-who-leaves","position":73,"of":88,"minutes":10,"answer":"When the thing you are predicting is rare, a model can be 95% accurate by never predicting it at all.","keyIdea":"When the thing you are predicting is rare, a model can be 95% accurate by never predicting it at all, so judge it by how many of its warnings were right and how many real cases it caught, at a threshold set by what the warning costs you."},"body":{"format":"markdown","text":"## The question, made specific\n\n\"Which customers are about to stop buying?\" is the most commonly attempted prediction in small business, and the most commonly misread.\n\nFirst it has to be made answerable. *Stop buying* is not a thing customers announce. You have to define it: a customer who ordered at least three times and has not ordered in 120 days, where 120 comes from your own data: look at the distribution of gaps between orders and pick a point well beyond where most repeat customers sit.\n\nThat definition is now the thing you are predicting. Change it and every number changes with it, which is why it goes in writing before anything is built.\n\n## The base rate decides how to read every figure\n\nSuppose 5% of your customers lapse in any quarter.\n\nA model that predicts \"nobody will lapse\", always, is 95% accurate. It is also completely worthless. This is the accuracy paradox, and it is why accuracy is close to useless as a measure of anything rare.\n\nTwo figures replace it, and both are ordinary questions in plain words.\n\n**Of the customers it flagged, how many actually lapsed?** That is precision, and it is the one that decides whether acting on the list is worth your time. Precision of 30% means two out of three people you contact were not going anywhere.\n\n**Of the customers who actually lapsed, how many did it flag?** That is recall, and it decides how much of the problem you are catching. Recall of 40% means three out of five walked away unflagged.\n\nYou cannot have both. Flag more people and recall rises while precision falls; flag fewer and the reverse. There is no setting that escapes this, and a vendor quoting one figure without the other is quoting the flattering half.\n\n> Figure: One hundred flagged customers, out of fifteen hundred. A two by two grid, rows The model flagged them and The model said nothing, columns They lapsed and They stayed: The model flagged them and They lapsed: 30 of the 75 who left, reached in time with a call and a voucher.; The model flagged them and They stayed: 70 calls to customers who were going nowhere.; The model said nothing and They lapsed: 45 of the 75 left without anybody noticing.; The model said nothing and They stayed: 1,355 customers, correctly left alone.. Five per cent lapse in a quarter, so a model that predicts nobody will lapse is ninety-five per cent accurate and worthless. Flag more and recall rises while precision falls; the threshold comes from what a call costs against what a retained customer is worth.\n\n## Setting the threshold from what the action costs\n\nA model outputs a score, not a decision. You choose the cut-off, and you choose it from arithmetic about your own business rather than from a default.\n\nSay a flagged customer gets a phone call and a 10% voucher. The call costs you ten minutes; the voucher costs margin only if they were coming back anyway. A retained customer is worth USD 400 over the next year.\n\nThen a wrong flag is cheap and a missed lapse is expensive, so you set the threshold low, accept precision around 25%, and catch more of them.\n\nReverse it (the intervention is an expensive hamper and a visit), and a wrong flag hurts, so you raise the threshold, contact fewer people, and accept that you will miss most lapses.\n\nThis is the whole of threshold-setting, and it is a business decision wearing a technical costume. Nobody but you can make it.\n\n## Before modelling: the rule that usually wins\n\nSort customers by days since last order, descending. Take the ones past your threshold who used to order regularly. That is your list.\n\nFor most small businesses this simple rule captures the large majority of what a model would find, costs nothing, and is explicable to whoever makes the calls. Build it first and measure it. Then, if you want, see whether a model beats it: using a holdout period, as in the previous lesson, because a model scored on the customers it learned from will look magnificent.\n\n## What actually moves the number\n\nNot the algorithm. The features: the columns you give it.\n\nRecency, frequency and monetary value, computed per customer, carry most of the signal in most businesses. Add gap-to-previous-gap: a customer whose intervals are lengthening is a useful signal and almost nobody computes it. Add whether their last experience involved a complaint, a delay or a substitution.\n\nAn hour spent building those columns is worth more than any amount of model selection, and this is the general rule in applied work: the data beats the method, nearly always.\n\n## The uncomfortable part\n\nA prediction about a person is a decision about how to treat them, and that deserves a moment's thought.\n\nDo not use a model like this to decide who gets worse service, slower responses, or a higher price. Beyond the fairness of it, which matters, there is a mechanical problem: the model learned from your past behaviour, so if you historically gave less attention to a group, the model will predict they were leaving and recommend you stop trying, and the prediction will come true because you acted on it.\n\nUse it to decide who gets *extra* attention. That way a wrong prediction costs you a phone call and gives somebody an unexpectedly good experience, which is the cheapest kind of error to make."},"check":{"question":"A shop's lapse model is reported as 93% accurate. Lapses run at about 6% of customers per quarter. The owner plans to act on the flagged list. What should she ask for first?","options":[{"text":"The confusion matrix broken down by customer value, since a model that is accurate overall may perform much worse on the high-value customers whose loss actually matters.","correct":false,"tempting":"A sharp follow-up question, and it presumes the headline figure is informative enough to break down."},{"text":"The share of flagged customers who actually lapsed, since 94% accuracy is available by flagging nobody.","correct":true},{"text":"Whether the accuracy was measured on a holdout period rather than the data the model learned from, since scores on seen data are inflated.","correct":false,"tempting":"The right instinct about holdouts, and a properly held-out 93% would still be uninformative here."},{"text":"The threshold the model is using, because accuracy varies with the cut-off and a different threshold would trade precision against recall for the same underlying model.","correct":false,"tempting":"Threshold-setting is the eventual decision, and it cannot be made sensibly while accuracy is the only figure on the table."}]},"prev":{"slug":"seasonality-and-the-events-register","title":"Patterns you can name, and the promotion that poisons the history","url":"https://addaly.com/learn/ai-for-small-business/seasonality-and-the-events-register"},"next":{"slug":"leakage-the-flattering-mistake","title":"The mistake that makes a model look brilliant","url":"https://addaly.com/learn/ai-for-small-business/leakage-the-flattering-mistake"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Predicting something rare, and why accuracy lies', in AI for a Small Business. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-for-small-business/predicting-who-leaves (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Predicting something rare, and why accuracy lies","courseTitle":"AI for a Small Business","module":{"index":8,"slug":"your-own-numbers","title":"Learning from your own numbers"},"answer":null,"keyIdea":"When the thing you are predicting is rare, a model can be 95% accurate by never predicting it at all, so judge it by how many of its warnings were right and how many real cases it caught, at a threshold set by what the warning costs you.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["They lapsed","They stayed"],"rows":["The model flagged them","The model said nothing"],"type":"matrix","cells":[["30 of the 75 who left, reached in time with a call and a voucher.","70 calls to customers who were going nowhere."],["45 of the 75 left without anybody noticing.","1,355 customers, correctly left alone."]],"title":"One hundred flagged customers, out of fifteen hundred","caption":"Five per cent lapse in a quarter, so a model that predicts nobody will lapse is ninety-five per cent accurate and worthless. Flag more and recall rises while precision falls; the threshold comes from what a call costs against what a retained customer is worth.","cellLabels":[["precision, 30 in 100","the price of a low threshold"],["recall, 30 of 75","most of the base"]]},"alt":"Figure: One hundred flagged customers, out of fifteen hundred. A two by two grid, rows The model flagged them and The model said nothing, columns They lapsed and They stayed: The model flagged them and They lapsed: 30 of the 75 who left, reached in time with a call and a voucher.; The model flagged them and They stayed: 70 calls to customers who were going nowhere.; The model said nothing and They lapsed: 45 of the 75 left without anybody noticing.; The model said nothing and They stayed: 1,355 customers, correctly left alone.. Five per cent lapse in a quarter, so a model that predicts nobody will lapse is ninety-five per cent accurate and worthless. Flag more and recall rises while precision falls; the threshold comes from what a call costs against what a retained customer is worth."}},"blocks":[{"t":"h2","id":"the-question-made-specific","runs":[{"text":"The question, made specific"}]},{"t":"p","runs":[{"text":"“Which customers are about to stop buying?” is the most commonly attempted prediction in small business, and the most commonly misread."}]},{"t":"p","runs":[{"text":"First it has to be made answerable. "},{"text":"Stop buying","i":true},{"text":" is not a thing customers announce. You have to define it: a customer who ordered at least three times and has not ordered in 120 days, where 120 comes from your own data: look at the distribution of gaps between orders and pick a point well beyond where most repeat customers sit."}]},{"t":"p","runs":[{"text":"That definition is now the thing you are predicting. Change it and every number changes with it, which is why it goes in writing before anything is built."}]},{"t":"h2","id":"the-base-rate-decides-how-to-read-every-figure","runs":[{"text":"The base rate decides how to read every figure"}]},{"t":"p","runs":[{"text":"Suppose 5% of your customers lapse in any quarter."}]},{"t":"p","runs":[{"text":"A model that predicts “nobody will lapse”, always, is 95% accurate. It is also completely worthless. This is the accuracy paradox, and it is why accuracy is close to useless as a measure of anything rare."}]},{"t":"p","runs":[{"text":"Two figures replace it, and both are ordinary questions in plain words."}]},{"t":"p","runs":[{"text":"Of the customers it flagged, how many actually lapsed?","b":true},{"text":" That is precision, and it is the one that decides whether acting on the list is worth your time. Precision of 30% means two out of three people you contact were not going anywhere."}]},{"t":"p","runs":[{"text":"Of the customers who actually lapsed, how many did it flag?","b":true},{"text":" That is recall, and it decides how much of the problem you are catching. Recall of 40% means three out of five walked away unflagged."}]},{"t":"p","runs":[{"text":"You cannot have both. Flag more people and recall rises while precision falls; flag fewer and the reverse. There is no setting that escapes this, and a vendor quoting one figure without the other is quoting the flattering half."}]},{"t":"h2","id":"setting-the-threshold-from-what-the-action-costs","runs":[{"text":"Setting the threshold from what the action costs"}]},{"t":"p","runs":[{"text":"A model outputs a score, not a decision. You choose the cut-off, and you choose it from arithmetic about your own business rather than from a default."}]},{"t":"p","runs":[{"text":"Say a flagged customer gets a phone call and a 10% voucher. The call costs you ten minutes; the voucher costs margin only if they were coming back anyway. A retained customer is worth USD 400 over the next year."}]},{"t":"p","runs":[{"text":"Then a wrong flag is cheap and a missed lapse is expensive, so you set the threshold low, accept precision around 25%, and catch more of them."}]},{"t":"p","runs":[{"text":"Reverse it (the intervention is an expensive hamper and a visit), and a wrong flag hurts, so you raise the threshold, contact fewer people, and accept that you will miss most lapses."}]},{"t":"p","runs":[{"text":"This is the whole of threshold-setting, and it is a business decision wearing a technical costume. Nobody but you can make it."}]},{"t":"h2","id":"before-modelling-the-rule-that-usually-wins","runs":[{"text":"Before modelling: the rule that usually wins"}]},{"t":"p","runs":[{"text":"Sort customers by days since last order, descending. Take the ones past your threshold who used to order regularly. That is your list."}]},{"t":"p","runs":[{"text":"For most small businesses this simple rule captures the large majority of what a model would find, costs nothing, and is explicable to whoever makes the calls. Build it first and measure it. Then, if you want, see whether a model beats it: using a holdout period, as in the previous lesson, because a model scored on the customers it learned from will look magnificent."}]},{"t":"h2","id":"what-actually-moves-the-number","runs":[{"text":"What actually moves the number"}]},{"t":"p","runs":[{"text":"Not the algorithm. The features: the columns you give it."}]},{"t":"p","runs":[{"text":"Recency, frequency and monetary value, computed per customer, carry most of the signal in most businesses. Add gap-to-previous-gap: a customer whose intervals are lengthening is a useful signal and almost nobody computes it. Add whether their last experience involved a complaint, a delay or a substitution."}]},{"t":"p","runs":[{"text":"An hour spent building those columns is worth more than any amount of model selection, and this is the general rule in applied work: the data beats the method, nearly always."}]},{"t":"h2","id":"the-uncomfortable-part","runs":[{"text":"The uncomfortable part"}]},{"t":"p","runs":[{"text":"A prediction about a person is a decision about how to treat them, and that deserves a moment’s thought."}]},{"t":"p","runs":[{"text":"Do not use a model like this to decide who gets worse service, slower responses, or a higher price. Beyond the fairness of it, which matters, there is a mechanical problem: the model learned from your past behaviour, so if you historically gave less attention to a group, the model will predict they were leaving and recommend you stop trying, and the prediction will come true because you acted on it."}]},{"t":"p","runs":[{"text":"Use it to decide who gets "},{"text":"extra","i":true},{"text":" attention. That way a wrong prediction costs you a phone call and gives somebody an unexpectedly good experience, which is the cheapest kind of error to make."}]}],"check":{"question":[{"text":"A shop’s lapse model is reported as 93% accurate. Lapses run at about 6% of customers per quarter. The owner plans to act on the flagged list. What should she ask for first?"}],"options":[{"text":[{"text":"The confusion matrix broken down by customer value, since a model that is accurate overall may perform much worse on the high-value customers whose loss actually matters."}],"correct":false,"tempting":[{"text":"A sharp follow-up question, and it presumes the headline figure is informative enough to break down."}]},{"text":[{"text":"The share of flagged customers who actually lapsed, since 94% accuracy is available by flagging nobody."}],"correct":true,"tempting":null},{"text":[{"text":"Whether the accuracy was measured on a holdout period rather than the data the model learned from, since scores on seen data are inflated."}],"correct":false,"tempting":[{"text":"The right instinct about holdouts, and a properly held-out 93% would still be uninformative here."}]},{"text":[{"text":"The threshold the model is using, because accuracy varies with the cut-off and a different threshold would trade precision against recall for the same underlying model."}],"correct":false,"tempting":[{"text":"Threshold-setting is the eventual decision, and it cannot be made sensibly while accuracy is the only figure on the table."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"seasonality-and-the-events-register","title":"Patterns you can name, and the promotion that poisons the history","minutes":8},"next":{"slug":"leakage-the-flattering-mistake","title":"The mistake that makes a model look brilliant","minutes":9}}}