Feature importance, and the ways it misleads

The number everybody prints

model.feature_importances_

One line, a satisfying bar chart, and a plausible story about what drives the outcome. It is also, in its default form, systematically biased in ways that are easy to demonstrate and rarely mentioned.

What impurity importance actually measures

For a tree ensemble, the default importance sums the impurity decrease each feature achieved across all splits, weighted by how many rows passed through those nodes.

That definition contains two biases.

Bias towards high-cardinality features. A feature with many distinct values offers many candidate thresholds, so by chance one of them will reduce impurity. Continuous features and high-cardinality categoricals therefore accumulate importance even when they carry no signal.

The demonstration is easy and worth running once. Add a column of random integers between 1 and 10,000 to your dataset and refit. It will appear in the top five features, above predictive binary columns. Nothing is broken; the metric is measuring what it says it measures, and what it measures is not what you wanted.

Computed on training data. Impurity importance reflects how the model used a feature to fit the training set, including the parts it overfitted. A feature that helped the model memorise noise scores highly.

Permutation importance

A better default. Shuffle one feature’s values in the validation set, breaking its relationship with the target while leaving the rest of the row intact, and measure how much the score drops.

from sklearn.inspection import permutation_importance
r = permutation_importance(model, X_val, y_val, n_repeats=10,
                           scoring="roc_auc", random_state=0)

This measures what you probably meant: how much this model, on held-out data, depends on this feature. It works for any model, it uses the metric you care about, and it gives a standard deviation across repeats so you can see which differences are real.

It has one serious failure mode, and it is not optional knowledge.

The correlated-feature problem

Suppose height_cm and height_inches are both in the model. Shuffle one, and the model reads the other and predicts fine. Score barely drops. Shuffle the other, same result. Both features get near-zero importance, and the conclusion “height does not matter” is exactly backwards.

This affects both methods, and it affects them in different directions: impurity importance splits the credit between correlated features, permutation importance zeroes both.

Two practical responses:

1. Cluster correlated features and permute the whole group at once. Compute the correlation matrix, cluster it, and shuffle each cluster together. This tells you what the group is worth, which is usually the question.
2. Drop-column importance. Remove a feature, refit, measure the drop. This is the most trustworthy method and costs one refit per feature, so it is affordable only with fast models and few columns.

The second also has a subtlety: if you drop height_cm and refit, the model learns to use height_inches instead and the score does not fall. The answer is the same: cluster first.

Importance is not effect, and not causation

Three distinctions that get collapsed constantly.

Importance is not effect size. “Income is the most important feature” does not tell you whether more income raises or lowers the prediction, or by how much. It says the model relies on it.

Importance is a property of the model, not the world. Two models fitted on the same data can rank features differently, both legitimately, because they use different structure.

Importance is not causation. Module 1 made this argument and it applies here with full force: a feature can be important because it is a consequence of the target. A high importance on a suspicious feature is a leakage alarm before it is anything else.

What importance is good for

Having spent the lesson on limits, the honest positive list.

• Finding leakage. One feature far above the others on a hard problem is the most reliable leakage detector there is.
• Feature selection. Drop the bottom half, refit, check whether the score moved. Often it does not, and you have halved your pipeline’s maintenance cost.
• Sanity checking. If your domain expert says the top feature should be irrelevant, one of you has learned something.
• Debugging a pipeline. A feature with exactly zero importance across every tree is often not reaching the model at all.

None of those uses requires importance to be a measure of real-world influence. All of them are valuable. The failure is only ever in the last step, when a bar chart becomes a slide titled “what drives customer churn”.