{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"from-one-tree-to-a-forest","title":"Bagging: many unstable models, averaged","url":"https://addaly.com/learn/machine-learning-foundations/from-one-tree-to-a-forest","position":55,"of":92,"minutes":9,"answer":"Bagging reduces variance by averaging, and a random forest adds per-split feature sampling because decorrelating the trees removes more variance than making each individual tree better.","keyIdea":"Bagging reduces variance by averaging, and a random forest adds per-split feature sampling because decorrelating the trees removes more variance than making each individual tree better."},"body":{"format":"markdown","text":"## Averaging cancels variance\n\nA single deep tree is high variance: fit it on a different sample and you get a different model. But its errors on any given row are somewhat random, and the average of many random errors is smaller than any one of them.\n\nThat is the whole idea of bagging: bootstrap aggregating. Fit the same high-variance model on many bootstrap samples of the data and average the predictions. Bias stays roughly where it was; variance falls.\n\n## The bootstrap sample\n\nDraw n rows from n rows, with replacement. Some rows appear two or three times, and about 36.8% appear not at all: the limit of `(1 - 1/n)^n` as n grows, which is `1/e`.\n\nThose left-out rows are called **out-of-bag**, and they are a free validation set. Every tree can be scored on the rows it did not see, and averaging that across trees gives you an estimate of generalisation error with no separate holdout and no cross-validation loop.\n\n```python\nRandomForestClassifier(n_estimators=500, oob_score=True).fit(X, y).oob_score_\n```\n\nThis is useful on small datasets, where every row you can avoid spending on validation matters.\n\n## What makes a forest more than bagged trees\n\nBagging alone leaves the trees too similar. If one feature is strongly predictive, every tree splits on it first, and the trees end up correlated, and averaging correlated things removes much less variance than averaging independent ones.\n\nRandom forests add a second source of randomness: **at each split, only a random subset of features is considered.**\n\n```python\nRandomForestClassifier(max_features=\"sqrt\")     # classification default\nRandomForestRegressor(max_features=1.0)         # regression default\n```\n\nWith 36 features and `max_features=\"sqrt\"`, each split chooses among 6 randomly selected features. Sometimes the dominant feature is not among them, so the tree is forced to find a different structure. The individual trees get slightly worse and much more different from each other, and the ensemble gets better. Breiman's insight, in 2001, was that decorrelating the trees matters more than making each one good.\n\n`max_features` is therefore the hyperparameter that actually matters in a forest. Lower means more decorrelation and more randomness; higher means stronger individual trees and more correlation.\n\n## What to tune, and what not to\n\n- **`n_estimators`**: not really a hyperparameter. More trees never make a forest worse, only slower; the score converges and then flattens. Set it to 300 to 500 and stop thinking about it. Do not tune it in a grid search.\n- **`max_features`**: tune this. Try `\"sqrt\"`, 0.3, 0.5, 1.0.\n- **`min_samples_leaf`**: tune this, especially with noisy data. 1 is the default and is often too aggressive; 5 or 20 frequently helps.\n- **`max_depth`**: usually leave unlimited. Deep trees are the point; averaging handles the variance.\n\nThat is four parameters of which two matter, which makes a forest one of the easiest models to get near its best performance with almost no tuning. It is an excellent second baseline after a linear model.\n\n## Extremely randomised trees\n\nOne more step in the same direction. `ExtraTreesClassifier` chooses split thresholds *at random* rather than searching for the best one, then keeps the best of those random candidates. Faster to train, since no threshold search happens, and more decorrelated still. On noisy data it often matches or beats a random forest, and it is worth trying because it costs one line.\n\n## Strengths and honest weaknesses\n\n**Strengths.** Almost no tuning. No scaling needed. Handles mixed types, non-linearity and interactions. Robust to outliers. Parallel across trees, so it uses every core. Gives out-of-bag error for free. Very hard to make catastrophically wrong.\n\n**Weaknesses, in order of how often they matter.**\n\n- **Model size.** 500 unlimited-depth trees on a million rows is hundreds of megabytes and slow to load. Gradient boosting with depth-6 trees is far smaller for similar accuracy.\n- **Prediction latency.** Traversing 500 trees per prediction is more work than one boosted model of 300 shallow trees.\n- **Usually slightly behind well-tuned boosting** on tabular accuracy, typically by a small margin.\n- **Regression predictions are conservative.** Averaging pulls extremes toward the middle, so a forest under-predicts the highest values and over-predicts the lowest. If the tails are what you care about, this is a real problem.\n- **No extrapolation**, inherited from trees.\n\n## When to reach for it\n\nWhen you want a strong result with almost no effort, when you have limited time to tune, when the data is noisy, or when you need out-of-bag error because the dataset is small. It is the model to fit second, after a linear baseline, before you decide whether tuning a boosted model is worth the afternoon."},"check":{"question":"A team sets max_features=1.0 in a random forest, so every split considers all features, and finds performance slightly worse than the sqrt default. Why?","options":[{"text":"Using all features overfits each tree, so the individual trees become too complex.","correct":false,"tempting":"Individual trees do become stronger, but a forest tolerates strong individual trees well; the loss comes from what happens between them, not within one."},{"text":"Considering all features at every split makes every tree pick the same dominant splits, so the trees become correlated and averaging removes less variance.","correct":true},{"text":"max_features=1.0 disables bootstrapping, so all trees see identical data.","correct":false,"tempting":"Bootstrapping is controlled by a separate parameter and remains active; the rows still differ between trees."},{"text":"Using all features slows training, so fewer trees complete within the same budget.","correct":false,"tempting":"Training is slower per split, but the number of trees is fixed by n_estimators rather than by a time budget."}]},"prev":{"slug":"how-a-tree-chooses-a-split","title":"How a tree chooses where to cut","url":"https://addaly.com/learn/machine-learning-foundations/how-a-tree-chooses-a-split"},"next":{"slug":"boosting-mechanics","title":"Boosting: each model fixes the last one's mistakes","url":"https://addaly.com/learn/machine-learning-foundations/boosting-mechanics"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Bagging: many unstable models, averaged', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/from-one-tree-to-a-forest (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Bagging: many unstable models, averaged","courseTitle":"Machine Learning, Foundations","module":{"index":6,"slug":"trees-and-ensembles","title":"Trees, ensembles, and the algorithms that win on tables"},"answer":null,"keyIdea":"Bagging reduces variance by averaging, and a random forest adds per-split feature sampling because decorrelating the trees removes more variance than making each individual tree better.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"averaging-cancels-variance","runs":[{"text":"Averaging cancels variance"}]},{"t":"p","runs":[{"text":"A single deep tree is high variance: fit it on a different sample and you get a different model. But its errors on any given row are somewhat random, and the average of many random errors is smaller than any one of them."}]},{"t":"p","runs":[{"text":"That is the whole idea of bagging: bootstrap aggregating. Fit the same high-variance model on many bootstrap samples of the data and average the predictions. Bias stays roughly where it was; variance falls."}]},{"t":"h2","id":"the-bootstrap-sample","runs":[{"text":"The bootstrap sample"}]},{"t":"p","runs":[{"text":"Draw n rows from n rows, with replacement. Some rows appear two or three times, and about 36.8% appear not at all: the limit of "},{"text":"(1 - 1/n)^n","code":true},{"text":" as n grows, which is "},{"text":"1/e","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"Those left-out rows are called "},{"text":"out-of-bag","b":true},{"text":", and they are a free validation set. Every tree can be scored on the rows it did not see, and averaging that across trees gives you an estimate of generalisation error with no separate holdout and no cross-validation loop."}]},{"t":"code","lang":"python","label":"Python","text":"RandomForestClassifier(n_estimators=500, oob_score=True).fit(X, y).oob_score_"},{"t":"p","runs":[{"text":"This is useful on small datasets, where every row you can avoid spending on validation matters."}]},{"t":"h2","id":"what-makes-a-forest-more-than-bagged-trees","runs":[{"text":"What makes a forest more than bagged trees"}]},{"t":"p","runs":[{"text":"Bagging alone leaves the trees too similar. If one feature is strongly predictive, every tree splits on it first, and the trees end up correlated, and averaging correlated things removes much less variance than averaging independent ones."}]},{"t":"p","runs":[{"text":"Random forests add a second source of randomness: "},{"text":"at each split, only a random subset of features is considered.","b":true}]},{"t":"code","lang":"python","label":"Python","text":"RandomForestClassifier(max_features=\"sqrt\")     # classification default\nRandomForestRegressor(max_features=1.0)         # regression default"},{"t":"p","runs":[{"text":"With 36 features and "},{"text":"max_features=\"sqrt\"","code":true},{"text":", each split chooses among 6 randomly selected features. Sometimes the dominant feature is not among them, so the tree is forced to find a different structure. The individual trees get slightly worse and much more different from each other, and the ensemble gets better. Breiman’s insight, in 2001, was that decorrelating the trees matters more than making each one good."}]},{"t":"p","runs":[{"text":"max_features","code":true},{"text":" is therefore the hyperparameter that actually matters in a forest. Lower means more decorrelation and more randomness; higher means stronger individual trees and more correlation."}]},{"t":"h2","id":"what-to-tune-and-what-not-to","runs":[{"text":"What to tune, and what not to"}]},{"t":"ul","items":[[{"text":"n_estimators","b":true,"code":true},{"text":": not really a hyperparameter. More trees never make a forest worse, only slower; the score converges and then flattens. Set it to 300 to 500 and stop thinking about it. Do not tune it in a grid search."}],[{"text":"max_features","b":true,"code":true},{"text":": tune this. Try "},{"text":"\"sqrt\"","code":true},{"text":", 0.3, 0.5, 1.0."}],[{"text":"min_samples_leaf","b":true,"code":true},{"text":": tune this, especially with noisy data. 1 is the default and is often too aggressive; 5 or 20 frequently helps."}],[{"text":"max_depth","b":true,"code":true},{"text":": usually leave unlimited. Deep trees are the point; averaging handles the variance."}]]},{"t":"p","runs":[{"text":"That is four parameters of which two matter, which makes a forest one of the easiest models to get near its best performance with almost no tuning. It is an excellent second baseline after a linear model."}]},{"t":"h2","id":"extremely-randomised-trees","runs":[{"text":"Extremely randomised trees"}]},{"t":"p","runs":[{"text":"One more step in the same direction. "},{"text":"ExtraTreesClassifier","code":true},{"text":" chooses split thresholds "},{"text":"at random","i":true},{"text":" rather than searching for the best one, then keeps the best of those random candidates. Faster to train, since no threshold search happens, and more decorrelated still. On noisy data it often matches or beats a random forest, and it is worth trying because it costs one line."}]},{"t":"h2","id":"strengths-and-honest-weaknesses","runs":[{"text":"Strengths and honest weaknesses"}]},{"t":"p","runs":[{"text":"Strengths.","b":true},{"text":" Almost no tuning. No scaling needed. Handles mixed types, non-linearity and interactions. Robust to outliers. Parallel across trees, so it uses every core. Gives out-of-bag error for free. Very hard to make catastrophically wrong."}]},{"t":"p","runs":[{"text":"Weaknesses, in order of how often they matter.","b":true}]},{"t":"ul","items":[[{"text":"Model size.","b":true},{"text":" 500 unlimited-depth trees on a million rows is hundreds of megabytes and slow to load. Gradient boosting with depth-6 trees is far smaller for similar accuracy."}],[{"text":"Prediction latency.","b":true},{"text":" Traversing 500 trees per prediction is more work than one boosted model of 300 shallow trees."}],[{"text":"Usually slightly behind well-tuned boosting","b":true},{"text":" on tabular accuracy, typically by a small margin."}],[{"text":"Regression predictions are conservative.","b":true},{"text":" Averaging pulls extremes toward the middle, so a forest under-predicts the highest values and over-predicts the lowest. If the tails are what you care about, this is a real problem."}],[{"text":"No extrapolation","b":true},{"text":", inherited from trees."}]]},{"t":"h2","id":"when-to-reach-for-it","runs":[{"text":"When to reach for it"}]},{"t":"p","runs":[{"text":"When you want a strong result with almost no effort, when you have limited time to tune, when the data is noisy, or when you need out-of-bag error because the dataset is small. It is the model to fit second, after a linear baseline, before you decide whether tuning a boosted model is worth the afternoon."}]}],"check":{"question":[{"text":"A team sets max_features=1.0 in a random forest, so every split considers all features, and finds performance slightly worse than the sqrt default. Why?"}],"options":[{"text":[{"text":"Using all features overfits each tree, so the individual trees become too complex."}],"correct":false,"tempting":[{"text":"Individual trees do become stronger, but a forest tolerates strong individual trees well; the loss comes from what happens between them, not within one."}]},{"text":[{"text":"Considering all features at every split makes every tree pick the same dominant splits, so the trees become correlated and averaging removes less variance."}],"correct":true,"tempting":null},{"text":[{"text":"max_features=1.0 disables bootstrapping, so all trees see identical data."}],"correct":false,"tempting":[{"text":"Bootstrapping is controlled by a separate parameter and remains active; the rows still differ between trees."}]},{"text":[{"text":"Using all features slows training, so fewer trees complete within the same budget."}],"correct":false,"tempting":[{"text":"Training is slower per split, but the number of trees is fixed by n_estimators rather than by a time budget."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"how-a-tree-chooses-a-split","title":"How a tree chooses where to cut","minutes":9},"next":{"slug":"boosting-mechanics","title":"Boosting: each model fixes the last one’s mistakes","minutes":9}}}