Turning categories into numbers without lying about them

The mistake that gets made first

Given a city column, the obvious move is to map cities to integers: Mumbai 0, Delhi 1, Chennai 2, Kolkata 3.

A linear model now believes Chennai is twice Delhi and that Mumbai and Kolkata are three units apart. Those statements are meaningless, but the model cannot know that; it will fit a coefficient as though the ordering were real. This is called label encoding, it is the default in a lot of tutorial code, and it is wrong for any unordered category fed to a linear or distance-based model.

It is not wrong for trees, which is the source of much of the confusion. A tree can split “city in {0, 3}” versus “city in {1, 2}” and recover an arbitrary grouping given enough splits. Label encoding for LightGBM is common practice; label encoding for logistic regression is a bug.

One-hot: the safe default

One column per category, 1 in the column for this row’s value and 0 elsewhere.

from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder(handle_unknown="ignore", sparse_output=True,
                    min_frequency=20)

Three arguments carry real weight.

handle_unknown="ignore" decides what happens when production sends a category that was not in the training data. Without it, the transform raises and your service returns an error. With it, that row gets all zeros across the block and a prediction still comes out. This is a genuine choice with a downside (silently encoding an unknown city as “none of the above” may be exactly wrong for your problem) but a crash at 2am is worse, and the alternative is at least deliberate.

min_frequency=20 folds every category with fewer than 20 rows into a single “infrequent” column. This is doing real work: a category with three examples cannot support a reliable coefficient, and giving it one invites the model to memorise those three rows.

sparse_output=True matters at scale. One-hot on 5,000 product codes gives 5,000 columns of which one is nonzero per row. Dense, a million rows would be 40 GB. Sparse, it is a few hundred megabytes. Linear models and SGD handle sparse matrices natively; some other estimators do not, and that is often the real reason to prefer another encoding.

Ordinal, where order is real

When the categories have an order (“never, rarely, sometimes, often, always”, or T-shirt sizes, or education level) integers are appropriate, but state the order explicitly rather than letting the encoder guess alphabetically:

from sklearn.preprocessing import OrdinalEncoder
OrdinalEncoder(categories=[["never", "rarely", "sometimes", "often", "always"]])

Remember what you have asserted: not just the order, but that the gaps are equal. “Never” to “rarely” gets the same numeric distance as “often” to “always”. Sometimes that is fine. When it is not, one-hot instead and let the model learn each level’s effect separately.

Target encoding, and the trap inside it

Replace each category with the mean of the target for that category. Mumbai becomes 0.043 because 4.3% of Mumbai rows defaulted.

This is powerful: one column instead of 5,000, and it carries exactly the information the model wants. It also leaks, in a way that produces spectacular validation scores and a model that fails in production.

The mechanism: a category with one row gets encoded as that row’s own target. The feature now contains the answer for that row. The model learns to trust it completely, and on unseen data where the encoding comes from other rows, that trust is unjustified.

The fixes are standard and non-optional:

1. Compute the encoding out-of-fold. For each training row, use the target mean computed from the other folds only. scikit-learn’s TargetEncoder does this internally with cross-fitting.
2. Smooth towards the global mean. A category with 4 rows should be pulled most of the way to the overall base rate; one with 40,000 rows should barely move. smooth="auto" handles it.

from sklearn.preprocessing import TargetEncoder
TargetEncoder(target_type="binary", smooth="auto", cv=5)

Even with both, treat a target-encoded model’s validation score with a little suspicion, and compare it against the same model with one-hot encoding before believing a large gain.

Counting, which nobody suspects

Replace each category with how often it appears. It uses no target information at all, so it cannot leak, it produces one column, and it is often surprisingly effective: for fraud, a rare merchant is a different kind of thing from a common one, and the count says so directly.

Cheap, safe, and underused. Try it before target encoding.

What to reach for

Under about 15 categories, one-hot and stop thinking about it. Between 15 and a few hundred, one-hot with min_frequency set, or count encoding. Above that, count encoding or a cross-fitted target encoder, and for gradient boosting specifically, CatBoost’s built-in categorical handling (which is an ordered target encoding done carefully) is frequently the best available option with no work at all.