Boosting: each model fixes the last one’s mistakes

The opposite construction

A forest fits many strong models in parallel and averages away their variance. Boosting fits many weak models in sequence, each one correcting what the previous ones got wrong, and reduces bias.

The trees in a boosted model are deliberately shallow: depth 3 to 6, sometimes a single split. On its own such a tree is barely better than guessing. Several hundred of them, each addressing the remaining error, is usually the strongest thing you can fit to a table.

Watching it work on five rows

Regression, targets [10, 12, 15, 9, 20].

Step 0. Predict the mean, 13.2, for everything. Residuals (what is left to explain) are [-3.2, -1.2, 1.8, -4.2, 6.8].

Step 1. Fit a small tree to the residuals, not to the original target. Suppose it learns to predict -2.9 for rows 1 and 4 and +2.0 for rows 3 and 5. Add a fraction of that (the learning rate, say 0.1) to the running prediction:

row 1: 13.2 + 0.1 * (-2.9) = 12.91
row 5: 13.2 + 0.1 * ( 2.0) = 13.40

Step 2. Recompute the residuals against the new predictions. Fit another small tree to those. Add a tenth of it. Repeat 500 times.

Each tree is fitted to what remains. That is gradient boosting in its original form, and the reason for the name is worth one sentence: for squared error, the residual is the negative gradient of the loss with respect to the prediction. So “fit the next model to the residuals” generalises to “fit the next model to the negative gradient”, which is what lets the same algorithm work with log loss, quantile loss, or any differentiable objective.

Why the small steps

The learning rate, 0.1, or 0.05, or 0.01, is why boosting works rather than overfitting immediately.

With a learning rate of 1.0 the model takes the full correction each time, drives the training residuals to zero in a few dozen trees, and has fitted the noise. With 0.05 it approaches the answer slowly, and each tree can only make a small claim, so noise gets averaged out across many trees while genuine signal accumulates.

The trade is exact and worth memorising: halve the learning rate, double the number of trees. Lower rates generalise slightly better and cost proportionally more time. 0.05 with 1,000 trees is a reasonable default; 0.3 with 100 trees is what you use while iterating quickly.

Early stopping is not optional

Unlike a random forest, a boosted model gets worse with too many trees. Each additional tree fits the remaining residuals, and once the signal is exhausted, what remains is noise.

So you stop when validation loss stops improving:

from sklearn.ensemble import HistGradientBoostingClassifier
m = HistGradientBoostingClassifier(
    learning_rate=0.05, max_iter=2000,
    early_stopping=True, validation_fraction=0.1, n_iter_no_change=50)
m.fit(X_tr, y_tr)
m.n_iter_     # how many trees it actually used

n_iter_no_change=50 means “keep going for 50 more rounds after the best score before giving up”, which stops you from halting on a noisy plateau. This one setting converts n_estimators from a parameter you must tune into one that tunes itself, and it is the single most useful line in this lesson.

Stochastic boosting

Subsample rows and columns for each tree, exactly as a forest does, and boosting improves: usually a small but consistent gain, plus a speedup.

subsample     = 0.8    # fraction of rows per tree
colsample_bytree = 0.8 # fraction of features per tree

The reason is the same as in bagging: randomness decorrelates the trees so their individual errors do not compound. In boosting it also acts as regularisation, because each tree sees less data and can therefore make weaker claims.

Boosting versus forests, plainly

Boosting is usually more accurate on tabular data, often by one to three points, and is what wins competitions.

Boosting is more sensitive. Get the learning rate or the depth wrong and it overfits badly, where a forest would have shrugged.

Boosting is sequential, so it cannot parallelise across trees the way a forest can. The libraries parallelise within a tree instead, over features and rows, which is why they are still fast.

Forests are more forgiving. If you have one afternoon and no time to tune, a forest with defaults is the safer bet. If you have two days, boosting will beat it.

AdaBoost, the original 1995 boosting algorithm, works slightly differently (it reweights misclassified rows rather than fitting residuals) and is now mostly of historical interest. The gradient formulation replaced it because it extends to any differentiable loss.