{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"boosting-mechanics","title":"Boosting: each model fixes the last one's mistakes","url":"https://addaly.com/learn/machine-learning-foundations/boosting-mechanics","position":56,"of":92,"minutes":9,"answer":"Boosting fits each shallow tree to the residual left by the previous ones and adds only a fraction of it, so accuracy accumulates slowly enough that noise averages out.","keyIdea":"Boosting fits each shallow tree to the residual left by the previous ones and adds only a fraction of it, so accuracy accumulates slowly enough that noise averages out, which is why a boosted model needs early stopping and a forest does not."},"body":{"format":"markdown","text":"## The opposite construction\n\nA forest fits many strong models in parallel and averages away their variance. Boosting fits many weak models in sequence, each one correcting what the previous ones got wrong, and reduces bias.\n\nThe trees in a boosted model are deliberately shallow: depth 3 to 6, sometimes a single split. On its own such a tree is barely better than guessing. Several hundred of them, each addressing the remaining error, is usually the strongest thing you can fit to a table.\n\n## Watching it work on five rows\n\nRegression, targets `[10, 12, 15, 9, 20]`.\n\n**Step 0.** Predict the mean, 13.2, for everything. Residuals (what is left to explain) are `[-3.2, -1.2, 1.8, -4.2, 6.8]`.\n\n**Step 1.** Fit a small tree *to the residuals*, not to the original target. Suppose it learns to predict `-2.9` for rows 1 and 4 and `+2.0` for rows 3 and 5. Add a fraction of that (the learning rate, say 0.1) to the running prediction:\n\n```text\nrow 1: 13.2 + 0.1 * (-2.9) = 12.91\nrow 5: 13.2 + 0.1 * ( 2.0) = 13.40\n```\n\n**Step 2.** Recompute the residuals against the new predictions. Fit another small tree to those. Add a tenth of it. Repeat 500 times.\n\nEach tree is fitted to what remains. That is gradient boosting in its original form, and the reason for the name is worth one sentence: for squared error, the residual *is* the negative gradient of the loss with respect to the prediction. So \"fit the next model to the residuals\" generalises to \"fit the next model to the negative gradient\", which is what lets the same algorithm work with log loss, quantile loss, or any differentiable objective.\n\n## Why the small steps\n\nThe learning rate, 0.1, or 0.05, or 0.01, is why boosting works rather than overfitting immediately.\n\nWith a learning rate of 1.0 the model takes the full correction each time, drives the training residuals to zero in a few dozen trees, and has fitted the noise. With 0.05 it approaches the answer slowly, and each tree can only make a small claim, so noise gets averaged out across many trees while genuine signal accumulates.\n\nThe trade is exact and worth memorising: **halve the learning rate, double the number of trees.** Lower rates generalise slightly better and cost proportionally more time. 0.05 with 1,000 trees is a reasonable default; 0.3 with 100 trees is what you use while iterating quickly.\n\n## Early stopping is not optional\n\nUnlike a random forest, **a boosted model gets worse with too many trees.** Each additional tree fits the remaining residuals, and once the signal is exhausted, what remains is noise.\n\n> Figure: Validation loss against number of trees, two learning rates. Line chart of Validation log loss against Trees: Learning rate 1.0 (0, 0.69) (25, 0.4) (50, 0.36) (100, 0.39) (200, 0.45) (400, 0.52) (1000, 0.62); Learning rate 0.05 (0, 0.69) (100, 0.52) (200, 0.42) (400, 0.35) (600, 0.32) (800, 0.31) (1000, 0.315). At a rate of 1.0 the model has fitted the residual noise by tree fifty and gets worse from there. At 0.05 it approaches a lower floor slowly and holds it, because each tree can only make a small claim. Halve the rate, double the trees, and let early stopping pick the count.\n\nSo you stop when validation loss stops improving:\n\n```python\nfrom sklearn.ensemble import HistGradientBoostingClassifier\nm = HistGradientBoostingClassifier(\n    learning_rate=0.05, max_iter=2000,\n    early_stopping=True, validation_fraction=0.1, n_iter_no_change=50)\nm.fit(X_tr, y_tr)\nm.n_iter_     # how many trees it actually used\n```\n\n`n_iter_no_change=50` means \"keep going for 50 more rounds after the best score before giving up\", which stops you from halting on a noisy plateau. This one setting converts `n_estimators` from a parameter you must tune into one that tunes itself, and it is the single most useful line in this lesson.\n\n## Stochastic boosting\n\nSubsample rows and columns for each tree, exactly as a forest does, and boosting improves: usually a small but consistent gain, plus a speedup.\n\n```text\nsubsample     = 0.8    # fraction of rows per tree\ncolsample_bytree = 0.8 # fraction of features per tree\n```\n\nThe reason is the same as in bagging: randomness decorrelates the trees so their individual errors do not compound. In boosting it also acts as regularisation, because each tree sees less data and can therefore make weaker claims.\n\n## Boosting versus forests, plainly\n\n**Boosting is usually more accurate** on tabular data, often by one to three points, and is what wins competitions.\n\n**Boosting is more sensitive.** Get the learning rate or the depth wrong and it overfits badly, where a forest would have shrugged.\n\n**Boosting is sequential**, so it cannot parallelise across trees the way a forest can. The libraries parallelise *within* a tree instead, over features and rows, which is why they are still fast.\n\n**Forests are more forgiving.** If you have one afternoon and no time to tune, a forest with defaults is the safer bet. If you have two days, boosting will beat it.\n\nAdaBoost, the original 1995 boosting algorithm, works slightly differently (it reweights misclassified rows rather than fitting residuals) and is now mostly of historical interest. The gradient formulation replaced it because it extends to any differentiable loss."},"check":{"question":"A team increases n_estimators from 500 to 5,000 in a gradient boosting model with learning_rate 0.1 and no early stopping. Validation loss gets worse. Why does this happen with boosting but not with a random forest?","options":[{"text":"Boosting trees are shallower, so more of them are needed and 5,000 is still insufficient to converge.","correct":false,"tempting":"Shallowness is why many trees are needed, but the observed degradation shows the model has passed its optimum rather than not reached it."},{"text":"Each boosted tree fits the residual left by the previous ones, so once the signal is exhausted the additional trees fit noise; forest trees are fitted independently and averaged, so extra trees only refine the average.","correct":true},{"text":"Boosting is sequential, so later trees are trained on progressively smaller subsamples and see too little data.","correct":false,"tempting":"Subsampling is optional and fixed per tree when used; later trees do not see progressively less data."},{"text":"The learning rate compounds across trees, so after enough rounds the predictions diverge numerically.","correct":false,"tempting":"Contributions are added with a fixed small weight rather than compounding, so predictions do not diverge; they overfit."}]},"prev":{"slug":"from-one-tree-to-a-forest","title":"Bagging: many unstable models, averaged","url":"https://addaly.com/learn/machine-learning-foundations/from-one-tree-to-a-forest"},"next":{"slug":"the-boosting-libraries","title":"XGBoost, LightGBM, CatBoost: what actually differs","url":"https://addaly.com/learn/machine-learning-foundations/the-boosting-libraries"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Boosting: each model fixes the last one's mistakes', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/boosting-mechanics (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Boosting: each model fixes the last one’s mistakes","courseTitle":"Machine Learning, Foundations","module":{"index":6,"slug":"trees-and-ensembles","title":"Trees, ensembles, and the algorithms that win on tables"},"answer":null,"keyIdea":"Boosting fits each shallow tree to the residual left by the previous ones and adds only a fraction of it, so accuracy accumulates slowly enough that noise averages out, which is why a boosted model needs early stopping and a forest does not.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"Validation loss against number of trees, two learning rates","series":[{"label":"Learning rate 1.0","points":[[0,0.69],[25,0.4],[50,0.36],[100,0.39],[200,0.45],[400,0.52],[1000,0.62]]},{"label":"Learning rate 0.05","points":[[0,0.69],[100,0.52],[200,0.42],[400,0.35],[600,0.32],[800,0.31],[1000,0.315]]}],"xLabel":"Trees","yLabel":"Validation log loss","caption":"At a rate of 1.0 the model has fitted the residual noise by tree fifty and gets worse from there. At 0.05 it approaches a lower floor slowly and holds it, because each tree can only make a small claim. Halve the rate, double the trees, and let early stopping pick the count."},"alt":"Figure: Validation loss against number of trees, two learning rates. Line chart of Validation log loss against Trees: Learning rate 1.0 (0, 0.69) (25, 0.4) (50, 0.36) (100, 0.39) (200, 0.45) (400, 0.52) (1000, 0.62); Learning rate 0.05 (0, 0.69) (100, 0.52) (200, 0.42) (400, 0.35) (600, 0.32) (800, 0.31) (1000, 0.315). At a rate of 1.0 the model has fitted the residual noise by tree fifty and gets worse from there. At 0.05 it approaches a lower floor slowly and holds it, because each tree can only make a small claim. Halve the rate, double the trees, and let early stopping pick the count."}},"blocks":[{"t":"h2","id":"the-opposite-construction","runs":[{"text":"The opposite construction"}]},{"t":"p","runs":[{"text":"A forest fits many strong models in parallel and averages away their variance. Boosting fits many weak models in sequence, each one correcting what the previous ones got wrong, and reduces bias."}]},{"t":"p","runs":[{"text":"The trees in a boosted model are deliberately shallow: depth 3 to 6, sometimes a single split. On its own such a tree is barely better than guessing. Several hundred of them, each addressing the remaining error, is usually the strongest thing you can fit to a table."}]},{"t":"h2","id":"watching-it-work-on-five-rows","runs":[{"text":"Watching it work on five rows"}]},{"t":"p","runs":[{"text":"Regression, targets "},{"text":"[10, 12, 15, 9, 20]","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"Step 0.","b":true},{"text":" Predict the mean, 13.2, for everything. Residuals (what is left to explain) are "},{"text":"[-3.2, -1.2, 1.8, -4.2, 6.8]","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"Step 1.","b":true},{"text":" Fit a small tree "},{"text":"to the residuals","i":true},{"text":", not to the original target. Suppose it learns to predict "},{"text":"-2.9","code":true},{"text":" for rows 1 and 4 and "},{"text":"+2.0","code":true},{"text":" for rows 3 and 5. Add a fraction of that (the learning rate, say 0.1) to the running prediction:"}]},{"t":"code","lang":"text","label":"Text","text":"row 1: 13.2 + 0.1 * (-2.9) = 12.91\nrow 5: 13.2 + 0.1 * ( 2.0) = 13.40"},{"t":"p","runs":[{"text":"Step 2.","b":true},{"text":" Recompute the residuals against the new predictions. Fit another small tree to those. Add a tenth of it. Repeat 500 times."}]},{"t":"p","runs":[{"text":"Each tree is fitted to what remains. That is gradient boosting in its original form, and the reason for the name is worth one sentence: for squared error, the residual "},{"text":"is","i":true},{"text":" the negative gradient of the loss with respect to the prediction. So “fit the next model to the residuals” generalises to “fit the next model to the negative gradient”, which is what lets the same algorithm work with log loss, quantile loss, or any differentiable objective."}]},{"t":"h2","id":"why-the-small-steps","runs":[{"text":"Why the small steps"}]},{"t":"p","runs":[{"text":"The learning rate, 0.1, or 0.05, or 0.01, is why boosting works rather than overfitting immediately."}]},{"t":"p","runs":[{"text":"With a learning rate of 1.0 the model takes the full correction each time, drives the training residuals to zero in a few dozen trees, and has fitted the noise. With 0.05 it approaches the answer slowly, and each tree can only make a small claim, so noise gets averaged out across many trees while genuine signal accumulates."}]},{"t":"p","runs":[{"text":"The trade is exact and worth memorising: "},{"text":"halve the learning rate, double the number of trees.","b":true},{"text":" Lower rates generalise slightly better and cost proportionally more time. 0.05 with 1,000 trees is a reasonable default; 0.3 with 100 trees is what you use while iterating quickly."}]},{"t":"h2","id":"early-stopping-is-not-optional","runs":[{"text":"Early stopping is not optional"}]},{"t":"p","runs":[{"text":"Unlike a random forest, "},{"text":"a boosted model gets worse with too many trees.","b":true},{"text":" Each additional tree fits the remaining residuals, and once the signal is exhausted, what remains is noise."}]},{"t":"p","runs":[{"text":"So you stop when validation loss stops improving:"}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.ensemble import HistGradientBoostingClassifier\nm = HistGradientBoostingClassifier(\n    learning_rate=0.05, max_iter=2000,\n    early_stopping=True, validation_fraction=0.1, n_iter_no_change=50)\nm.fit(X_tr, y_tr)\nm.n_iter_     # how many trees it actually used"},{"t":"p","runs":[{"text":"n_iter_no_change=50","code":true},{"text":" means “keep going for 50 more rounds after the best score before giving up”, which stops you from halting on a noisy plateau. This one setting converts "},{"text":"n_estimators","code":true},{"text":" from a parameter you must tune into one that tunes itself, and it is the single most useful line in this lesson."}]},{"t":"h2","id":"stochastic-boosting","runs":[{"text":"Stochastic boosting"}]},{"t":"p","runs":[{"text":"Subsample rows and columns for each tree, exactly as a forest does, and boosting improves: usually a small but consistent gain, plus a speedup."}]},{"t":"code","lang":"text","label":"Text","text":"subsample     = 0.8    # fraction of rows per tree\ncolsample_bytree = 0.8 # fraction of features per tree"},{"t":"p","runs":[{"text":"The reason is the same as in bagging: randomness decorrelates the trees so their individual errors do not compound. In boosting it also acts as regularisation, because each tree sees less data and can therefore make weaker claims."}]},{"t":"h2","id":"boosting-versus-forests-plainly","runs":[{"text":"Boosting versus forests, plainly"}]},{"t":"p","runs":[{"text":"Boosting is usually more accurate","b":true},{"text":" on tabular data, often by one to three points, and is what wins competitions."}]},{"t":"p","runs":[{"text":"Boosting is more sensitive.","b":true},{"text":" Get the learning rate or the depth wrong and it overfits badly, where a forest would have shrugged."}]},{"t":"p","runs":[{"text":"Boosting is sequential","b":true},{"text":", so it cannot parallelise across trees the way a forest can. The libraries parallelise "},{"text":"within","i":true},{"text":" a tree instead, over features and rows, which is why they are still fast."}]},{"t":"p","runs":[{"text":"Forests are more forgiving.","b":true},{"text":" If you have one afternoon and no time to tune, a forest with defaults is the safer bet. If you have two days, boosting will beat it."}]},{"t":"p","runs":[{"text":"AdaBoost, the original 1995 boosting algorithm, works slightly differently (it reweights misclassified rows rather than fitting residuals) and is now mostly of historical interest. The gradient formulation replaced it because it extends to any differentiable loss."}]}],"check":{"question":[{"text":"A team increases n_estimators from 500 to 5,000 in a gradient boosting model with learning_rate 0.1 and no early stopping. Validation loss gets worse. Why does this happen with boosting but not with a random forest?"}],"options":[{"text":[{"text":"Boosting trees are shallower, so more of them are needed and 5,000 is still insufficient to converge."}],"correct":false,"tempting":[{"text":"Shallowness is why many trees are needed, but the observed degradation shows the model has passed its optimum rather than not reached it."}]},{"text":[{"text":"Each boosted tree fits the residual left by the previous ones, so once the signal is exhausted the additional trees fit noise; forest trees are fitted independently and averaged, so extra trees only refine the average."}],"correct":true,"tempting":null},{"text":[{"text":"Boosting is sequential, so later trees are trained on progressively smaller subsamples and see too little data."}],"correct":false,"tempting":[{"text":"Subsampling is optional and fixed per tree when used; later trees do not see progressively less data."}]},{"text":[{"text":"The learning rate compounds across trees, so after enough rounds the predictions diverge numerically."}],"correct":false,"tempting":[{"text":"Contributions are added with a fixed small weight rather than compounding, so predictions do not diverge; they overfit."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"from-one-tree-to-a-forest","title":"Bagging: many unstable models, averaged","minutes":9},"next":{"slug":"the-boosting-libraries","title":"XGBoost, LightGBM, CatBoost: what actually differs","minutes":9}}}