Bagging: many unstable models, averaged

Averaging cancels variance

A single deep tree is high variance: fit it on a different sample and you get a different model. But its errors on any given row are somewhat random, and the average of many random errors is smaller than any one of them.

That is the whole idea of bagging: bootstrap aggregating. Fit the same high-variance model on many bootstrap samples of the data and average the predictions. Bias stays roughly where it was; variance falls.

The bootstrap sample

Draw n rows from n rows, with replacement. Some rows appear two or three times, and about 36.8% appear not at all: the limit of (1 - 1/n)^n as n grows, which is 1/e.

Those left-out rows are called out-of-bag, and they are a free validation set. Every tree can be scored on the rows it did not see, and averaging that across trees gives you an estimate of generalisation error with no separate holdout and no cross-validation loop.

RandomForestClassifier(n_estimators=500, oob_score=True).fit(X, y).oob_score_

This is useful on small datasets, where every row you can avoid spending on validation matters.

What makes a forest more than bagged trees

Bagging alone leaves the trees too similar. If one feature is strongly predictive, every tree splits on it first, and the trees end up correlated, and averaging correlated things removes much less variance than averaging independent ones.

Random forests add a second source of randomness: at each split, only a random subset of features is considered.

RandomForestClassifier(max_features="sqrt")     # classification default
RandomForestRegressor(max_features=1.0)         # regression default

With 36 features and max_features="sqrt", each split chooses among 6 randomly selected features. Sometimes the dominant feature is not among them, so the tree is forced to find a different structure. The individual trees get slightly worse and much more different from each other, and the ensemble gets better. Breiman’s insight, in 2001, was that decorrelating the trees matters more than making each one good.

max_features is therefore the hyperparameter that actually matters in a forest. Lower means more decorrelation and more randomness; higher means stronger individual trees and more correlation.

What to tune, and what not to

• n_estimators: not really a hyperparameter. More trees never make a forest worse, only slower; the score converges and then flattens. Set it to 300 to 500 and stop thinking about it. Do not tune it in a grid search.
• max_features: tune this. Try "sqrt", 0.3, 0.5, 1.0.
• min_samples_leaf: tune this, especially with noisy data. 1 is the default and is often too aggressive; 5 or 20 frequently helps.
• max_depth: usually leave unlimited. Deep trees are the point; averaging handles the variance.

That is four parameters of which two matter, which makes a forest one of the easiest models to get near its best performance with almost no tuning. It is an excellent second baseline after a linear model.

Extremely randomised trees

One more step in the same direction. ExtraTreesClassifier chooses split thresholds at random rather than searching for the best one, then keeps the best of those random candidates. Faster to train, since no threshold search happens, and more decorrelated still. On noisy data it often matches or beats a random forest, and it is worth trying because it costs one line.

Strengths and honest weaknesses

Strengths. Almost no tuning. No scaling needed. Handles mixed types, non-linearity and interactions. Robust to outliers. Parallel across trees, so it uses every core. Gives out-of-bag error for free. Very hard to make catastrophically wrong.

Weaknesses, in order of how often they matter.

• Model size. 500 unlimited-depth trees on a million rows is hundreds of megabytes and slow to load. Gradient boosting with depth-6 trees is far smaller for similar accuracy.
• Prediction latency. Traversing 500 trees per prediction is more work than one boosted model of 300 shallow trees.
• Usually slightly behind well-tuned boosting on tabular accuracy, typically by a small margin.
• Regression predictions are conservative. Averaging pulls extremes toward the middle, so a forest under-predicts the highest values and over-predicts the lowest. If the tails are what you care about, this is a real problem.
• No extrapolation, inherited from trees.

When to reach for it

When you want a strong result with almost no effort, when you have limited time to tune, when the data is noisy, or when you need out-of-bag error because the dataset is small. It is the model to fit second, after a linear baseline, before you decide whether tuning a boosted model is worth the afternoon.