{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"prediction-is-not-explanation","title":"Why a model that predicts well cannot tell you what to change","url":"https://addaly.com/learn/machine-learning-foundations/prediction-is-not-explanation","position":6,"of":92,"minutes":9,"answer":"A predictive model reports associations that hold when you leave the world alone; changing something requires an experiment or an explicit causal assumption, and no accuracy score can substitute for either.","keyIdea":"A predictive model reports associations that hold when you leave the world alone; changing something requires an experiment or an explicit causal assumption, and no accuracy score can substitute for either."},"body":{"format":"markdown","text":"## Two questions that look like one\n\n*Who will churn next month?* and *what should we do to stop them churning?* sound like the same question with an extra step. They are not, and a model built for the first cannot answer the second: not with better features, not with SHAP, not with more data.\n\nThe difference is that prediction only requires a correlation that holds. Intervention requires knowing what happens when you reach in and change something, which is a claim about a world that has not happened yet.\n\n## The example everyone should carry\n\nA telecom company finds that customers who call support are far more likely to churn. It is a strong signal, stable across two years and every region. As a predictor it works.\n\nNow read it as an instruction: *reduce churn by making it harder to call support.* Obviously absurd, and the absurdity comes from the same place every time: the call did not cause the churn, the underlying problem caused both. Remove the phone line and you remove the signal while leaving the cause untouched. Churn goes up, because now the frustrated customers cannot even complain.\n\nThe model was never wrong. It answered the question it was asked. Somebody then asked it a different question and read the old answer.\n\n## Why the coefficient does not mean what you want\n\nA linear model gives you a number per feature, and the temptation to read it causally is nearly irresistible. Here is the mechanism that breaks it.\n\nSuppose hospital patients on a certain drug die more often. Fit a model, and the drug's coefficient is positive. But the drug is given to the sickest patients. The coefficient is measuring \"was sick enough to be prescribed this\", and it will do so faithfully.\n\nYou might say: add severity as a feature and the drug's coefficient will clean up. Sometimes. This is called controlling for a confounder and it is the standard move, but it only works if you measured the confounder, measured it well, and there are no others. In practice severity is a doctor's judgement compressed into a code, and the part of it that is not in your data stays attached to the drug coefficient.\n\nWorse, controlling for the wrong thing actively creates bias. If you control for a variable that sits *downstream* of the treatment (a collider) you can manufacture an association out of nothing. This is not an edge case; it is the single most common statistical error in observational analysis, and it is invisible in any accuracy score.\n\n## The one thing that settles it\n\nA randomised experiment. Assign the change at random, and the randomisation guarantees that the treated and untreated groups differ only by chance in *everything*, measured or not, including the confounders you never thought of. That is what randomisation buys, and it is why it remains the gold standard after a century of attempts to replace it.\n\nWhen you cannot randomise (you cannot randomly assign smoking, or randomly deny a loan to test the model), there is a serious toolkit: difference-in-differences, instrumental variables, regression discontinuity, propensity matching, and the do-calculus that Judea Pearl's group formalised. Every one of these buys a causal claim by making an assumption you cannot test with your data. They are not tricks; they are honest bargains, and you should be able to state the assumption out loud before using one.\n\nThis course does not teach them. It teaches you to notice when you need them, which is the part people miss.\n\n## What to do on Monday\n\nWhen someone hands you a predictive model and asks what to change, three moves are available and all of them are legitimate.\n\n1. **Use the model for targeting, not for policy.** \"Predict who will churn, then have a human call them\" is fine. The model chose who to call; it did not claim the call works. Whether the call works is a separate experiment.\n2. **Run the experiment on the intervention.** Take the top 5,000 predicted churners, randomly give half of them the retention offer, measure the difference. Now you have a causal number about the offer, and the model's only job was to make the experiment cheaper by concentrating the population.\n3. **Say the honest sentence.** \"This tells us where the risk is. It does not tell us what to do about it, and if we act on the coefficients we may make things worse.\" This sentence has saved more money than most model improvements.\n\n## The limitation, stated plainly\n\nExplanation tools (SHAP values, permutation importance, partial dependence plots) are explanations *of the model*, not of the world. A SHAP value tells you how much this model's output moved because of this feature. That is useful for debugging, for spotting leakage, and for satisfying a regulator who asks why an application was declined. It is not an estimate of what would happen if you changed the feature.\n\nThe distinction sounds pedantic until the day a team acts on a SHAP plot, spends a quarter on the intervention it suggested, and measures nothing. Most people meet this once. The ones who read this lesson meet it in a lesson instead."},"check":{"question":"A retention model shows that customers who use the mobile app churn much less. The team proposes pushing every customer to install the app. What is the strongest reason to doubt the plan before running it?","options":[{"text":"The relationship may run the other way or come from a common cause (engaged customers both install the app and stay) so installing it for the disengaged need not change anything.","correct":true},{"text":"The app-usage feature is probably leaking, since usage is recorded after the churn window closes.","correct":false,"tempting":"Names a real hazard, but leakage would invalidate the prediction itself; here the prediction can be perfectly valid and the intervention still fail."},{"text":"The model has not been tested for fairness across customer segments, so the policy could be discriminatory.","correct":false,"tempting":"Raises an important separate concern that does not address whether the proposed intervention would work at all."},{"text":"The feature's coefficient has not been checked for statistical significance, so the association may be noise.","correct":false,"tempting":"Treats it as a sampling-uncertainty problem; the association can be large, stable and significant and the causal conclusion still be wrong."}]},"prev":{"slug":"the-baseline-you-must-beat","title":"The number your model has to beat before anyone should care","url":"https://addaly.com/learn/machine-learning-foundations/the-baseline-you-must-beat"},"next":{"slug":"when-not-to-use-machine-learning","title":"The problems where the right answer is a rule","url":"https://addaly.com/learn/machine-learning-foundations/when-not-to-use-machine-learning"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Why a model that predicts well cannot tell you what to change', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/prediction-is-not-explanation (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Why a model that predicts well cannot tell you what to change","courseTitle":"Machine Learning, Foundations","module":{"index":1,"slug":"what-learning-is","title":"What learning is, and what it is not"},"answer":null,"keyIdea":"A predictive model reports associations that hold when you leave the world alone; changing something requires an experiment or an explicit causal assumption, and no accuracy score can substitute for either.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"two-questions-that-look-like-one","runs":[{"text":"Two questions that look like one"}]},{"t":"p","runs":[{"text":"Who will churn next month?","i":true},{"text":" and "},{"text":"what should we do to stop them churning?","i":true},{"text":" sound like the same question with an extra step. They are not, and a model built for the first cannot answer the second: not with better features, not with SHAP, not with more data."}]},{"t":"p","runs":[{"text":"The difference is that prediction only requires a correlation that holds. Intervention requires knowing what happens when you reach in and change something, which is a claim about a world that has not happened yet."}]},{"t":"h2","id":"the-example-everyone-should-carry","runs":[{"text":"The example everyone should carry"}]},{"t":"p","runs":[{"text":"A telecom company finds that customers who call support are far more likely to churn. It is a strong signal, stable across two years and every region. As a predictor it works."}]},{"t":"p","runs":[{"text":"Now read it as an instruction: "},{"text":"reduce churn by making it harder to call support.","i":true},{"text":" Obviously absurd, and the absurdity comes from the same place every time: the call did not cause the churn, the underlying problem caused both. Remove the phone line and you remove the signal while leaving the cause untouched. Churn goes up, because now the frustrated customers cannot even complain."}]},{"t":"p","runs":[{"text":"The model was never wrong. It answered the question it was asked. Somebody then asked it a different question and read the old answer."}]},{"t":"h2","id":"why-the-coefficient-does-not-mean-what-you-want","runs":[{"text":"Why the coefficient does not mean what you want"}]},{"t":"p","runs":[{"text":"A linear model gives you a number per feature, and the temptation to read it causally is nearly irresistible. Here is the mechanism that breaks it."}]},{"t":"p","runs":[{"text":"Suppose hospital patients on a certain drug die more often. Fit a model, and the drug’s coefficient is positive. But the drug is given to the sickest patients. The coefficient is measuring “was sick enough to be prescribed this”, and it will do so faithfully."}]},{"t":"p","runs":[{"text":"You might say: add severity as a feature and the drug’s coefficient will clean up. Sometimes. This is called controlling for a confounder and it is the standard move, but it only works if you measured the confounder, measured it well, and there are no others. In practice severity is a doctor’s judgement compressed into a code, and the part of it that is not in your data stays attached to the drug coefficient."}]},{"t":"p","runs":[{"text":"Worse, controlling for the wrong thing actively creates bias. If you control for a variable that sits "},{"text":"downstream","i":true},{"text":" of the treatment (a collider) you can manufacture an association out of nothing. This is not an edge case; it is the single most common statistical error in observational analysis, and it is invisible in any accuracy score."}]},{"t":"h2","id":"the-one-thing-that-settles-it","runs":[{"text":"The one thing that settles it"}]},{"t":"p","runs":[{"text":"A randomised experiment. Assign the change at random, and the randomisation guarantees that the treated and untreated groups differ only by chance in "},{"text":"everything","i":true},{"text":", measured or not, including the confounders you never thought of. That is what randomisation buys, and it is why it remains the gold standard after a century of attempts to replace it."}]},{"t":"p","runs":[{"text":"When you cannot randomise (you cannot randomly assign smoking, or randomly deny a loan to test the model), there is a serious toolkit: difference-in-differences, instrumental variables, regression discontinuity, propensity matching, and the do-calculus that Judea Pearl’s group formalised. Every one of these buys a causal claim by making an assumption you cannot test with your data. They are not tricks; they are honest bargains, and you should be able to state the assumption out loud before using one."}]},{"t":"p","runs":[{"text":"This course does not teach them. It teaches you to notice when you need them, which is the part people miss."}]},{"t":"h2","id":"what-to-do-on-monday","runs":[{"text":"What to do on Monday"}]},{"t":"p","runs":[{"text":"When someone hands you a predictive model and asks what to change, three moves are available and all of them are legitimate."}]},{"t":"ol","start":1,"items":[[{"text":"Use the model for targeting, not for policy.","b":true},{"text":" “Predict who will churn, then have a human call them” is fine. The model chose who to call; it did not claim the call works. Whether the call works is a separate experiment."}],[{"text":"Run the experiment on the intervention.","b":true},{"text":" Take the top 5,000 predicted churners, randomly give half of them the retention offer, measure the difference. Now you have a causal number about the offer, and the model’s only job was to make the experiment cheaper by concentrating the population."}],[{"text":"Say the honest sentence.","b":true},{"text":" “This tells us where the risk is. It does not tell us what to do about it, and if we act on the coefficients we may make things worse.” This sentence has saved more money than most model improvements."}]]},{"t":"h2","id":"the-limitation-stated-plainly","runs":[{"text":"The limitation, stated plainly"}],"role":"limit"},{"t":"p","runs":[{"text":"Explanation tools (SHAP values, permutation importance, partial dependence plots) are explanations "},{"text":"of the model","i":true},{"text":", not of the world. A SHAP value tells you how much this model’s output moved because of this feature. That is useful for debugging, for spotting leakage, and for satisfying a regulator who asks why an application was declined. It is not an estimate of what would happen if you changed the feature."}]},{"t":"p","runs":[{"text":"The distinction sounds pedantic until the day a team acts on a SHAP plot, spends a quarter on the intervention it suggested, and measures nothing. Most people meet this once. The ones who read this lesson meet it in a lesson instead."}]}],"check":{"question":[{"text":"A retention model shows that customers who use the mobile app churn much less. The team proposes pushing every customer to install the app. What is the strongest reason to doubt the plan before running it?"}],"options":[{"text":[{"text":"The relationship may run the other way or come from a common cause (engaged customers both install the app and stay) so installing it for the disengaged need not change anything."}],"correct":true,"tempting":null},{"text":[{"text":"The app-usage feature is probably leaking, since usage is recorded after the churn window closes."}],"correct":false,"tempting":[{"text":"Names a real hazard, but leakage would invalidate the prediction itself; here the prediction can be perfectly valid and the intervention still fail."}]},{"text":[{"text":"The model has not been tested for fairness across customer segments, so the policy could be discriminatory."}],"correct":false,"tempting":[{"text":"Raises an important separate concern that does not address whether the proposed intervention would work at all."}]},{"text":[{"text":"The feature’s coefficient has not been checked for statistical significance, so the association may be noise."}],"correct":false,"tempting":[{"text":"Treats it as a sampling-uncertainty problem; the association can be large, stable and significant and the causal conclusion still be wrong."}]}],"order":[2,3,1,0],"why":null},"prev":{"slug":"the-baseline-you-must-beat","title":"The number your model has to beat before anyone should care","minutes":8},"next":{"slug":"when-not-to-use-machine-learning","title":"The problems where the right answer is a rule","minutes":8}}}