XGBoost, LightGBM, CatBoost: what actually differs

Three libraries, one algorithm

All three implement gradient boosted trees. All three are free and open source. The differences are real but narrower than the marketing suggests, and on a well-tuned problem they usually land within a point of each other.

XGBoost (2014) is the one that made the technique famous. Mature, enormously well documented, level-wise tree growth by default, and available in every language you might need. It is the safe institutional choice.

LightGBM (2017) is usually the fastest and lightest on memory. Two design choices explain most of that: it bins continuous features into a histogram (typically 255 buckets) before searching for splits, which turns a sort into a count; and it grows trees leaf-wise rather than level-wise, always splitting the leaf that promises the biggest gain. Leaf-wise growth gets more accuracy per tree and overfits more readily on small data, which is why num_leaves matters more there than max_depth.

CatBoost (2017) handles categorical features natively with an ordered target statistic: a target encoding computed using only rows that came earlier in a random permutation, which avoids the leakage from module 3 by construction. It also builds symmetric trees, where every node at a level uses the same split, which makes prediction extremely fast. Its defaults are unusually good; it is often the best result you can get without tuning at all.

What to reach for

• Many categorical columns, or you want a strong result with no tuning: CatBoost.
• Large data, and training time matters: LightGBM.
• You want maximum ecosystem support, or your organisation already uses it: XGBoost.
• You do not want another dependency: HistGradientBoostingClassifier in scikit-learn, which is a LightGBM-style histogram implementation, handles NaN natively, and is close enough to the others that the difference rarely decides anything.

That last option deserves emphasis. It is in the library you already have, it needs no install, and it is a competitive model.

The parameters that matter

Every library exposes forty or more. Six of them do the work.

1. learning_rate: 0.05 for a final model, 0.3 while iterating. Everything else is easier to tune once this is settled.

2. n_estimators with early stopping: set it high, let early stopping choose. Do not put it in a grid search.

3. Depth: max_depth 4 to 8 for XGBoost and CatBoost; num_leaves for LightGBM, where the rough correspondence is num_leaves around 2^max_depth but usually smaller. LightGBM’s default of 31 is often too high for datasets under 10,000 rows.

4. min_child_weight / min_data_in_leaf: the minimum evidence required to make a split. Raise it when overfitting. This is the most under-used control of the six.

5. subsample and colsample_bytree: 0.8 each is a good default and rarely a bad one.

6. reg_lambda: L2 on the leaf values. Raise it on noisy data.

A workable starting point:

import lightgbm as lgb
m = lgb.LGBMClassifier(
    learning_rate=0.05, n_estimators=3000, num_leaves=31,
    min_child_samples=30, subsample=0.8, subsample_freq=1,
    colsample_bytree=0.8, reg_lambda=1.0, random_state=0)
m.fit(X_tr, y_tr, eval_set=[(X_val, y_val)],
      callbacks=[lgb.early_stopping(100)])

Note subsample_freq=1: in LightGBM, setting subsample alone does nothing without it, which is a long-standing trap that costs people an afternoon.

Where these models are strong

On tabular data, boosted trees remain the state of the art. This has been tested carefully, most notably by Grinsztajn and colleagues in 2022, who compared tree ensembles against several neural architectures across many tabular datasets and found trees ahead on most of them, while also finding the gap narrows as datasets grow beyond a few tens of thousands of rows.

The reasons they give are mechanical rather than mysterious: tabular features are often uninformative individually, columns have no meaningful ordering for a network to exploit, and target functions are frequently irregular in a way piecewise-constant models handle naturally. That is a live research area rather than a settled matter, and the honest summary is that on a table with fewer than 100,000 rows, boosted trees are the model to beat and usually are not beaten.

Where they are weak

• Extrapolation: inherited from trees, unchanged by ensembling.
• Very high-dimensional sparse data: a hundred thousand TF-IDF columns is a linear model’s territory, not a tree’s.
• Images, audio, raw text: no.
• Latency at scale: 1,000 trees per prediction adds up, though the fix is well trodden: fewer, deeper trees, or a compiled predictor such as Treelite.
• Explainability: 1,000 trees is not readable. Which is the next two lessons.