Machine Learning, Foundations · Chapter 06
Each model fixes the last one's mistakes
1:59 · Narrated with a synthetic voice. Captions are available in the player.
Read the transcript
On tables, a linear model is your floor and gradient boosting is usually your answer. A random forest fits many strong trees in parallel and averages their variance away. Boosting does the opposite. Many deliberately shallow trees in sequence, each one fitted to what the previous ones got wrong, and only a fraction of each correction is added to the running prediction. That fraction is the learning rate, and it is why boosting accumulates accuracy instead of memorising immediately. Halve the learning rate and double the number of trees; the trade is exact. And unlike a forest, a boosted model gets worse with too many trees, because once the signal is used up the remaining residual is noise. Early stopping is not optional here. Everybody prints the feature importance bar chart, and in its default form it is biased. Impurity importance rewards a column for offering many candidate thresholds, so a column of random integers between one and ten thousand will land in your top five. Permutation importance on held-out data measures what you probably meant. But both collapse when two columns carry the same information: shuffle one, the model reads the other, the score barely moves, and both look worthless. Cluster the correlated columns and permute the group together. And keep the standing rule in view. Importance describes the model, not the world, and a suspicious spike is a leakage alarm before it is anything else.