{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"the-boosting-libraries","title":"XGBoost, LightGBM, CatBoost: what actually differs","url":"https://addaly.com/learn/machine-learning-foundations/the-boosting-libraries","position":57,"of":92,"minutes":9,"answer":"The three boosting libraries implement the same algorithm with different tree-growth and categorical handling, and six hyperparameters (learning rate, early stopping, depth, minimum leaf evidence, subsampling and L2) cover essentially all the tuning that matters.","keyIdea":"The three boosting libraries implement the same algorithm with different tree-growth and categorical handling, and six hyperparameters (learning rate, early stopping, depth, minimum leaf evidence, subsampling and L2) cover essentially all the tuning that matters."},"body":{"format":"markdown","text":"## Three libraries, one algorithm\n\nAll three implement gradient boosted trees. All three are free and open source. The differences are real but narrower than the marketing suggests, and on a well-tuned problem they usually land within a point of each other.\n\n**XGBoost** (2014) is the one that made the technique famous. Mature, enormously well documented, level-wise tree growth by default, and available in every language you might need. It is the safe institutional choice.\n\n**LightGBM** (2017) is usually the fastest and lightest on memory. Two design choices explain most of that: it bins continuous features into a histogram (typically 255 buckets) before searching for splits, which turns a sort into a count; and it grows trees **leaf-wise** rather than level-wise, always splitting the leaf that promises the biggest gain. Leaf-wise growth gets more accuracy per tree and overfits more readily on small data, which is why `num_leaves` matters more there than `max_depth`.\n\n**CatBoost** (2017) handles categorical features natively with an ordered target statistic: a target encoding computed using only rows that came earlier in a random permutation, which avoids the leakage from module 3 by construction. It also builds symmetric trees, where every node at a level uses the same split, which makes prediction extremely fast. Its defaults are unusually good; it is often the best result you can get without tuning at all.\n\n## What to reach for\n\n- Many categorical columns, or you want a strong result with no tuning: **CatBoost**.\n- Large data, and training time matters: **LightGBM**.\n- You want maximum ecosystem support, or your organisation already uses it: **XGBoost**.\n- You do not want another dependency: **`HistGradientBoostingClassifier`** in scikit-learn, which is a LightGBM-style histogram implementation, handles NaN natively, and is close enough to the others that the difference rarely decides anything.\n\nThat last option deserves emphasis. It is in the library you already have, it needs no install, and it is a competitive model.\n\n## The parameters that matter\n\nEvery library exposes forty or more. Six of them do the work.\n\n**1. `learning_rate`**: 0.05 for a final model, 0.3 while iterating. Everything else is easier to tune once this is settled.\n\n**2. `n_estimators` with early stopping**: set it high, let early stopping choose. Do not put it in a grid search.\n\n**3. Depth**: `max_depth` 4 to 8 for XGBoost and CatBoost; `num_leaves` for LightGBM, where the rough correspondence is `num_leaves` around `2^max_depth` but usually smaller. LightGBM's default of 31 is often too high for datasets under 10,000 rows.\n\n**4. `min_child_weight` / `min_data_in_leaf`**: the minimum evidence required to make a split. Raise it when overfitting. This is the most under-used control of the six.\n\n**5. `subsample` and `colsample_bytree`**: 0.8 each is a good default and rarely a bad one.\n\n**6. `reg_lambda`**: L2 on the leaf values. Raise it on noisy data.\n\nA workable starting point:\n\n```python\nimport lightgbm as lgb\nm = lgb.LGBMClassifier(\n    learning_rate=0.05, n_estimators=3000, num_leaves=31,\n    min_child_samples=30, subsample=0.8, subsample_freq=1,\n    colsample_bytree=0.8, reg_lambda=1.0, random_state=0)\nm.fit(X_tr, y_tr, eval_set=[(X_val, y_val)],\n      callbacks=[lgb.early_stopping(100)])\n```\n\nNote `subsample_freq=1`: in LightGBM, setting `subsample` alone does nothing without it, which is a long-standing trap that costs people an afternoon.\n\n## Where these models are strong\n\nOn tabular data, boosted trees remain the state of the art. This has been tested carefully, most notably by Grinsztajn and colleagues in 2022, who compared tree ensembles against several neural architectures across many tabular datasets and found trees ahead on most of them, while also finding the gap narrows as datasets grow beyond a few tens of thousands of rows.\n\nThe reasons they give are mechanical rather than mysterious: tabular features are often uninformative individually, columns have no meaningful ordering for a network to exploit, and target functions are frequently irregular in a way piecewise-constant models handle naturally. That is a live research area rather than a settled matter, and the honest summary is that on a table with fewer than 100,000 rows, boosted trees are the model to beat and usually are not beaten.\n\n## Where they are weak\n\n- **Extrapolation**: inherited from trees, unchanged by ensembling.\n- **Very high-dimensional sparse data**: a hundred thousand TF-IDF columns is a linear model's territory, not a tree's.\n- **Images, audio, raw text**: no.\n- **Latency at scale**: 1,000 trees per prediction adds up, though the fix is well trodden: fewer, deeper trees, or a compiled predictor such as Treelite.\n- **Explainability**: 1,000 trees is not readable. Which is the next two lessons."},"check":{"question":"A team moves from XGBoost to LightGBM on a 6,000-row dataset with identical hyperparameters and finds LightGBM overfits noticeably more. What is the most likely cause?","options":[{"text":"LightGBM bins continuous features into histograms, which loses resolution and forces the model to memorise bin boundaries.","correct":false,"tempting":"Histogram binning is a real difference and it slightly reduces resolution, which tends to regularise rather than to overfit."},{"text":"LightGBM's categorical handling applies target statistics that leak on small datasets.","correct":false,"tempting":"Describes CatBoost's mechanism rather than LightGBM's, and LightGBM's categorical support is not enabled by identical numeric hyperparameters."},{"text":"LightGBM grows trees leaf-wise, always splitting the highest-gain leaf, so at the default num_leaves of 31 it builds deeper, more specialised trees than level-wise growth does on 6,000 rows.","correct":true},{"text":"LightGBM defaults to a higher learning rate, so the same n_estimators produces a more aggressive fit.","correct":false,"tempting":"The premise states identical hyperparameters, so the learning rate is the same in both."}]},"prev":{"slug":"boosting-mechanics","title":"Boosting: each model fixes the last one's mistakes","url":"https://addaly.com/learn/machine-learning-foundations/boosting-mechanics"},"next":{"slug":"nearest-neighbours","title":"k-nearest neighbours, and why distance stops meaning anything","url":"https://addaly.com/learn/machine-learning-foundations/nearest-neighbours"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'XGBoost, LightGBM, CatBoost: what actually differs', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/the-boosting-libraries (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"XGBoost, LightGBM, CatBoost: what actually differs","courseTitle":"Machine Learning, Foundations","module":{"index":6,"slug":"trees-and-ensembles","title":"Trees, ensembles, and the algorithms that win on tables"},"answer":null,"keyIdea":"The three boosting libraries implement the same algorithm with different tree-growth and categorical handling, and six hyperparameters (learning rate, early stopping, depth, minimum leaf evidence, subsampling and L2) cover essentially all the tuning that matters.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"three-libraries-one-algorithm","runs":[{"text":"Three libraries, one algorithm"}]},{"t":"p","runs":[{"text":"All three implement gradient boosted trees. All three are free and open source. The differences are real but narrower than the marketing suggests, and on a well-tuned problem they usually land within a point of each other."}]},{"t":"p","runs":[{"text":"XGBoost","b":true},{"text":" (2014) is the one that made the technique famous. Mature, enormously well documented, level-wise tree growth by default, and available in every language you might need. It is the safe institutional choice."}]},{"t":"p","runs":[{"text":"LightGBM","b":true},{"text":" (2017) is usually the fastest and lightest on memory. Two design choices explain most of that: it bins continuous features into a histogram (typically 255 buckets) before searching for splits, which turns a sort into a count; and it grows trees "},{"text":"leaf-wise","b":true},{"text":" rather than level-wise, always splitting the leaf that promises the biggest gain. Leaf-wise growth gets more accuracy per tree and overfits more readily on small data, which is why "},{"text":"num_leaves","code":true},{"text":" matters more there than "},{"text":"max_depth","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"CatBoost","b":true},{"text":" (2017) handles categorical features natively with an ordered target statistic: a target encoding computed using only rows that came earlier in a random permutation, which avoids the leakage from module 3 by construction. It also builds symmetric trees, where every node at a level uses the same split, which makes prediction extremely fast. Its defaults are unusually good; it is often the best result you can get without tuning at all."}]},{"t":"h2","id":"what-to-reach-for","runs":[{"text":"What to reach for"}]},{"t":"ul","items":[[{"text":"Many categorical columns, or you want a strong result with no tuning: "},{"text":"CatBoost","b":true},{"text":"."}],[{"text":"Large data, and training time matters: "},{"text":"LightGBM","b":true},{"text":"."}],[{"text":"You want maximum ecosystem support, or your organisation already uses it: "},{"text":"XGBoost","b":true},{"text":"."}],[{"text":"You do not want another dependency: "},{"text":"HistGradientBoostingClassifier","b":true,"code":true},{"text":" in scikit-learn, which is a LightGBM-style histogram implementation, handles NaN natively, and is close enough to the others that the difference rarely decides anything."}]]},{"t":"p","runs":[{"text":"That last option deserves emphasis. It is in the library you already have, it needs no install, and it is a competitive model."}]},{"t":"h2","id":"the-parameters-that-matter","runs":[{"text":"The parameters that matter"}]},{"t":"p","runs":[{"text":"Every library exposes forty or more. Six of them do the work."}]},{"t":"p","runs":[{"text":"1. ","b":true},{"text":"learning_rate","b":true,"code":true},{"text":": 0.05 for a final model, 0.3 while iterating. Everything else is easier to tune once this is settled."}]},{"t":"p","runs":[{"text":"2. ","b":true},{"text":"n_estimators","b":true,"code":true},{"text":" with early stopping","b":true},{"text":": set it high, let early stopping choose. Do not put it in a grid search."}]},{"t":"p","runs":[{"text":"3. Depth","b":true},{"text":": "},{"text":"max_depth","code":true},{"text":" 4 to 8 for XGBoost and CatBoost; "},{"text":"num_leaves","code":true},{"text":" for LightGBM, where the rough correspondence is "},{"text":"num_leaves","code":true},{"text":" around "},{"text":"2^max_depth","code":true},{"text":" but usually smaller. LightGBM’s default of 31 is often too high for datasets under 10,000 rows."}]},{"t":"p","runs":[{"text":"4. ","b":true},{"text":"min_child_weight","b":true,"code":true},{"text":" / ","b":true},{"text":"min_data_in_leaf","b":true,"code":true},{"text":": the minimum evidence required to make a split. Raise it when overfitting. This is the most under-used control of the six."}]},{"t":"p","runs":[{"text":"5. ","b":true},{"text":"subsample","b":true,"code":true},{"text":" and ","b":true},{"text":"colsample_bytree","b":true,"code":true},{"text":": 0.8 each is a good default and rarely a bad one."}]},{"t":"p","runs":[{"text":"6. ","b":true},{"text":"reg_lambda","b":true,"code":true},{"text":": L2 on the leaf values. Raise it on noisy data."}]},{"t":"p","runs":[{"text":"A workable starting point:"}]},{"t":"code","lang":"python","label":"Python","text":"import lightgbm as lgb\nm = lgb.LGBMClassifier(\n    learning_rate=0.05, n_estimators=3000, num_leaves=31,\n    min_child_samples=30, subsample=0.8, subsample_freq=1,\n    colsample_bytree=0.8, reg_lambda=1.0, random_state=0)\nm.fit(X_tr, y_tr, eval_set=[(X_val, y_val)],\n      callbacks=[lgb.early_stopping(100)])"},{"t":"p","runs":[{"text":"Note "},{"text":"subsample_freq=1","code":true},{"text":": in LightGBM, setting "},{"text":"subsample","code":true},{"text":" alone does nothing without it, which is a long-standing trap that costs people an afternoon."}]},{"t":"h2","id":"where-these-models-are-strong","runs":[{"text":"Where these models are strong"}]},{"t":"p","runs":[{"text":"On tabular data, boosted trees remain the state of the art. This has been tested carefully, most notably by Grinsztajn and colleagues in 2022, who compared tree ensembles against several neural architectures across many tabular datasets and found trees ahead on most of them, while also finding the gap narrows as datasets grow beyond a few tens of thousands of rows."}]},{"t":"p","runs":[{"text":"The reasons they give are mechanical rather than mysterious: tabular features are often uninformative individually, columns have no meaningful ordering for a network to exploit, and target functions are frequently irregular in a way piecewise-constant models handle naturally. That is a live research area rather than a settled matter, and the honest summary is that on a table with fewer than 100,000 rows, boosted trees are the model to beat and usually are not beaten."}]},{"t":"h2","id":"where-they-are-weak","runs":[{"text":"Where they are weak"}]},{"t":"ul","items":[[{"text":"Extrapolation","b":true},{"text":": inherited from trees, unchanged by ensembling."}],[{"text":"Very high-dimensional sparse data","b":true},{"text":": a hundred thousand TF-IDF columns is a linear model’s territory, not a tree’s."}],[{"text":"Images, audio, raw text","b":true},{"text":": no."}],[{"text":"Latency at scale","b":true},{"text":": 1,000 trees per prediction adds up, though the fix is well trodden: fewer, deeper trees, or a compiled predictor such as Treelite."}],[{"text":"Explainability","b":true},{"text":": 1,000 trees is not readable. Which is the next two lessons."}]]}],"check":{"question":[{"text":"A team moves from XGBoost to LightGBM on a 6,000-row dataset with identical hyperparameters and finds LightGBM overfits noticeably more. What is the most likely cause?"}],"options":[{"text":[{"text":"LightGBM bins continuous features into histograms, which loses resolution and forces the model to memorise bin boundaries."}],"correct":false,"tempting":[{"text":"Histogram binning is a real difference and it slightly reduces resolution, which tends to regularise rather than to overfit."}]},{"text":[{"text":"LightGBM’s categorical handling applies target statistics that leak on small datasets."}],"correct":false,"tempting":[{"text":"Describes CatBoost’s mechanism rather than LightGBM’s, and LightGBM’s categorical support is not enabled by identical numeric hyperparameters."}]},{"text":[{"text":"LightGBM grows trees leaf-wise, always splitting the highest-gain leaf, so at the default num_leaves of 31 it builds deeper, more specialised trees than level-wise growth does on 6,000 rows."}],"correct":true,"tempting":null},{"text":[{"text":"LightGBM defaults to a higher learning rate, so the same n_estimators produces a more aggressive fit."}],"correct":false,"tempting":[{"text":"The premise states identical hyperparameters, so the learning rate is the same in both."}]}],"order":[3,1,2,0],"why":null},"prev":{"slug":"boosting-mechanics","title":"Boosting: each model fixes the last one’s mistakes","minutes":9},"next":{"slug":"nearest-neighbours","title":"k-nearest neighbours, and why distance stops meaning anything","minutes":8}}}