When a column has thousands of levels

The shape of the problem

User ID. Product SKU. Postcode. Merchant name. IP address. URL path. These columns behave differently from city, and the difference is not just size.

They follow a long tail: a small number of levels cover most rows, and a very large number of levels appear once or twice. In a typical e-commerce dataset, the top 200 products might be 60% of orders and the remaining 80,000 products the other 40%, most of them with under five rows each.

That shape breaks the standard advice. One-hot gives you 80,000 columns of which most carry four examples: enough for the model to memorise, not enough for it to learn anything transferable.

Four workable approaches

1. Keep the head, group the tail. Take the categories covering 95% of rows and one-hot them; everything else becomes other. Crude, fast, and it captures most of what is there. min_frequency in OneHotEncoder does exactly this.

2. Count or frequency encoding. One column, no leakage, and for many of these fields the frequency is the useful signal. A postcode that appears in 40,000 orders and one that appears in three are different in a way that matters.

3. The hashing trick. Apply a hash function to the category string and take it modulo a fixed number of columns, say 2^18.

from sklearn.feature_extraction import FeatureHasher
h = FeatureHasher(n_features=2**18, input_type="string")

This is fixed-size regardless of how many distinct values arrive, needs no vocabulary stored, handles unseen categories automatically, and streams: you never need the whole dataset in memory. Collisions happen: two categories land in the same column and their effects get mixed. With 2^18 columns and 80,000 categories, collisions are rare enough that the loss is usually small.

The real cost is interpretability. Column 149,203 corresponds to nothing you can name, so you can never explain what the model learned. For ad-click prediction at web scale this has been standard for over a decade. For a credit decision that must be explained, it is unusable.

4. Learned embeddings. Give each category a small vector (8 to 64 numbers) and learn those numbers as part of training. This is what neural networks do with categorical inputs, and it is the technique behind word vectors and recommender systems.

The advantage over one-hot is that similar categories can end up with similar vectors, so the model transfers what it learned about one to another. Two postcodes with similar customers get similar embeddings and share statistical strength. One-hot cannot express similarity at all; every category is equidistant from every other.

The cost is that you need enough data per category to learn the vector, and enough infrastructure to train a network. Module 7 treats embeddings properly.

A rule for the dimension

The widely used heuristic for embedding size is the fourth root of the number of categories, times something between 1 and 3. For 80,000 categories that gives roughly 17 to 50. It is not theory, it is a rule of thumb that people found works, and you should tune around it rather than trust it.

The identifier question

Should user_id be a feature at all?

If you will predict for the same users you trained on (a recommender for an existing customer base) then yes, and an embedding per user is the whole design of a recommender system.

If you will predict for users you have never seen (new applicants, new visitors) then no. The model will learn per-user quirks that do not exist for anybody new, your random split will hide this because the same users appear on both sides, and performance will be far worse in production than in testing. This is the single most common cause of the “it worked in the notebook” failure.

The test is simple and worth writing on a sticky note: will this exact value exist at prediction time for a row I have never seen? If not, it is not a feature; it is an identifier, and its place is in the index.

What to do with unseen levels

Whatever encoding you choose, production will send you a value that was not in training. Decide now, in code, what happens:

• Hashing handles it for free: the new value hashes somewhere.
• Count encoding can map it to zero or to the minimum observed count.
• Target encoding maps it to the global mean, which is the sensible default.
• One-hot with handle_unknown="ignore" gives an all-zero block.

And log it. A rising rate of unseen categories is one of the earliest and clearest signals that your data has drifted, and it is far easier to monitor than the drift in a continuous feature. Module 9 comes back to this.