Why a model that predicts well cannot tell you what to change

Two questions that look like one

Who will churn next month? and what should we do to stop them churning? sound like the same question with an extra step. They are not, and a model built for the first cannot answer the second: not with better features, not with SHAP, not with more data.

The difference is that prediction only requires a correlation that holds. Intervention requires knowing what happens when you reach in and change something, which is a claim about a world that has not happened yet.

The example everyone should carry

A telecom company finds that customers who call support are far more likely to churn. It is a strong signal, stable across two years and every region. As a predictor it works.

Now read it as an instruction: reduce churn by making it harder to call support. Obviously absurd, and the absurdity comes from the same place every time: the call did not cause the churn, the underlying problem caused both. Remove the phone line and you remove the signal while leaving the cause untouched. Churn goes up, because now the frustrated customers cannot even complain.

The model was never wrong. It answered the question it was asked. Somebody then asked it a different question and read the old answer.

Why the coefficient does not mean what you want

A linear model gives you a number per feature, and the temptation to read it causally is nearly irresistible. Here is the mechanism that breaks it.

Suppose hospital patients on a certain drug die more often. Fit a model, and the drug’s coefficient is positive. But the drug is given to the sickest patients. The coefficient is measuring “was sick enough to be prescribed this”, and it will do so faithfully.

You might say: add severity as a feature and the drug’s coefficient will clean up. Sometimes. This is called controlling for a confounder and it is the standard move, but it only works if you measured the confounder, measured it well, and there are no others. In practice severity is a doctor’s judgement compressed into a code, and the part of it that is not in your data stays attached to the drug coefficient.

Worse, controlling for the wrong thing actively creates bias. If you control for a variable that sits downstream of the treatment (a collider) you can manufacture an association out of nothing. This is not an edge case; it is the single most common statistical error in observational analysis, and it is invisible in any accuracy score.

The one thing that settles it

A randomised experiment. Assign the change at random, and the randomisation guarantees that the treated and untreated groups differ only by chance in everything, measured or not, including the confounders you never thought of. That is what randomisation buys, and it is why it remains the gold standard after a century of attempts to replace it.

When you cannot randomise (you cannot randomly assign smoking, or randomly deny a loan to test the model), there is a serious toolkit: difference-in-differences, instrumental variables, regression discontinuity, propensity matching, and the do-calculus that Judea Pearl’s group formalised. Every one of these buys a causal claim by making an assumption you cannot test with your data. They are not tricks; they are honest bargains, and you should be able to state the assumption out loud before using one.

This course does not teach them. It teaches you to notice when you need them, which is the part people miss.

What to do on Monday

When someone hands you a predictive model and asks what to change, three moves are available and all of them are legitimate.

1. Use the model for targeting, not for policy. “Predict who will churn, then have a human call them” is fine. The model chose who to call; it did not claim the call works. Whether the call works is a separate experiment.
2. Run the experiment on the intervention. Take the top 5,000 predicted churners, randomly give half of them the retention offer, measure the difference. Now you have a causal number about the offer, and the model’s only job was to make the experiment cheaper by concentrating the population.
3. Say the honest sentence. “This tells us where the risk is. It does not tell us what to do about it, and if we act on the coefficients we may make things worse.” This sentence has saved more money than most model improvements.

The limitation, stated plainly

Explanation tools (SHAP values, permutation importance, partial dependence plots) are explanations of the model, not of the world. A SHAP value tells you how much this model’s output moved because of this feature. That is useful for debugging, for spotting leakage, and for satisfying a regulator who asks why an application was declined. It is not an estimate of what would happen if you changed the feature.

The distinction sounds pedantic until the day a team acts on a SHAP plot, spends a quarter on the intervention it suggested, and measures nothing. Most people meet this once. The ones who read this lesson meet it in a lesson instead.