Explaining one prediction, and the shape of one feature

Two different questions

“Which features does this model rely on overall?” was the last lesson. Two more specific questions come up constantly and need different tools.

**Why did this application get declined?** A local explanation, for one row.

**How does the prediction change as this feature changes?** A global shape, for one column.

SHAP: dividing the credit for one prediction

SHAP assigns each feature a number saying how much it pushed this particular prediction away from the average prediction. The numbers add up exactly:

base value (average prediction)          0.12
+ income = 240,000                      -0.04
+ late_payments = 3                     +0.19
+ account_age_days = 45                 +0.08
+ everything else                       -0.02
= this prediction                        0.33

That is a readable answer to “why 0.33”, and it is the format regulators tend to accept for adverse action notices.

The underlying idea comes from cooperative game theory. Shapley values answer: if features are players cooperating to produce a prediction, what is each one’s fair share of the result? The fair division is defined by averaging a feature’s marginal contribution over every possible order in which features could be added. That is the only allocation satisfying a small set of reasonable axioms, which is why SHAP has a stronger theoretical footing than most explanation methods.

Computing it exactly requires evaluating all 2^n subsets. TreeSHAP does it exactly for tree ensembles in polynomial time, which is why SHAP became standard alongside gradient boosting specifically.

import shap
explainer = shap.TreeExplainer(model)
sv = explainer.shap_values(X_val)
shap.summary_plot(sv, X_val)

What SHAP does not tell you

Three limits, stated because they are routinely ignored.

It explains the model, not the world. A SHAP value of +0.19 for late_payments = 3 means this model’s output was 0.19 higher because of that value. It does not mean reducing late payments by three would lower the person’s real risk. Module 1’s distinction, again.

Correlated features share credit in ways that depend on the method. With height_cm and height_inches in the model, how the credit divides between them is partly an artefact of the algorithm’s assumptions rather than a fact about the model.

The standard approximation assumes feature independence. When features are dependent, the method evaluates the model on combinations that never occur in reality: an income of ₹40 lakh with a postcode where nobody earns that. The model’s output on impossible inputs then influences the attribution. There are variants that condition on the data distribution instead, and they are slower and less used.

Partial dependence: the shape of one feature

A different question. Take every row, set income to 200,000 for all of them, average the predictions. Repeat for 250,000, 300,000, and so on. Plot the curve.

from sklearn.inspection import PartialDependenceDisplay
PartialDependenceDisplay.from_estimator(model, X_val, ["income", "age"])

The result shows the average relationship the model learned. It is frequently the most informative single plot you can produce about a tree ensemble, and it regularly reveals things nobody expected: a threshold effect at a round number that turns out to be a policy rule, a relationship that reverses direction, a plateau where a linear model assumed a slope.

Its flaw is the same independence assumption. If income and postcode are correlated, setting income to 40 lakh for every row creates impossible people, and the average includes the model’s behaviour on them.

ALE plots (accumulated local effects) fix this by computing the effect locally, using only rows where the feature is already near the value being examined. They are more trustworthy on correlated data and less widely used. The alibi library implements them, free.

ICE: the curves the average hides

Partial dependence plots an average, and averages hide heterogeneity. An individual conditional expectation plot draws one line per row instead.

If all the lines are parallel, the average is a fair summary. If half slope up and half slope down, the average is flat and completely misleading: there is an interaction, and the effect of this feature depends on something else in the row.

PartialDependenceDisplay.from_estimator(model, X_val, ["income"],
                                        kind="both")

kind="both" draws the ICE lines and the average together. It costs nothing and it is the check that stops you reporting a flat average over two opposite populations.

Choosing between them

• One decision to justify to a person: SHAP.
• Understanding the model’s overall behaviour on one feature: partial dependence, with ICE lines.
• Which features matter at all: permutation importance, from the previous lesson.
• Correlated features and you need it right: ALE, and read the caveats.

And the standing rule underneath all four: these describe the model. If somebody is going to act on the explanation to change the world, that requires an experiment, and no amount of attribution substitutes for one.