Missing values, and the information in the gap

Why it is missing changes what to do

Three situations, with consequences that differ enough to be worth the vocabulary.

Missing completely at random. A sensor dropped packets. Nothing about the row determines whether the value is there. Rare in practice, and the only case where dropping rows is safe.

Missing at random, given what you observed. Income is missing more often for younger applicants, but among applicants of the same age, whether income is recorded has nothing to do with the income itself. Imputation using the other columns is principled here.

Missing not at random. The value is missing because of what it is. People with very high or very low incomes decline to state them. Patients who felt fine did not return for the follow-up test. Here no imputation can recover the truth, because the information needed to reconstruct it is exactly what is absent.

The uncomfortable part: you cannot distinguish these from the data. It is a judgement about the process that produced the rows, and it comes from talking to whoever collected them.

The gap is often the best feature

The practical response to all three cases starts the same way: add an indicator column saying whether the value was missing, then impute.

from sklearn.impute import SimpleImputer
SimpleImputer(strategy="median", add_indicator=True)

That one argument is the highest-value thing in this lesson. In real datasets, missingness is frequently more predictive than the value would have been. A loan applicant who left employer blank, a patient with no recorded blood pressure, a form where the optional field is empty: each carries information about the person and the process, and imputing without the flag throws it away permanently.

Worth being clear-eyed about what that means: your model may be learning from an administrative artefact rather than anything about the world. That can be fine, or it can be a fairness problem if missingness correlates with a protected group, or it can break the moment a form changes. Know which you have.

Choosing an imputation strategy

• Median for numeric. Robust to skew, and most real numeric columns are skewed. Mean is fine for symmetric data and dragged around by outliers otherwise.
• Most frequent or an explicit "missing" category for categoricals. The explicit category is usually better: it lets the model treat missingness as its own level, which is the categorical version of the indicator column.
• Constant, out of range (a value like -999) works for tree models specifically, because a tree can isolate it with one split. It is nonsense for linear models, which will treat -999 as an enormous negative quantity.
• KNNImputer fills from the k most similar rows. More accurate, considerably slower, and it needs scaled features to compute similarity sensibly.
• IterativeImputer models each column from the others in turn. Powerful and slow, and easy to over-trust: a well-imputed value is still a guess presented with the same confidence as a measurement.

Let the model handle it, when it can

Some models handle missingness natively, and better than imputation:

from sklearn.ensemble import HistGradientBoostingClassifier
HistGradientBoostingClassifier()   # accepts NaN directly

At each split, it sends missing values whichever way reduces the loss more, learned separately per split. XGBoost and LightGBM do the same. This is better than imputing, because the model gets to decide what missing means in each context rather than having one substituted value everywhere.

If you are using boosted trees, the correct answer to missing values is often to do nothing.

When to drop

• A column that is 95% missing is usually not worth keeping, although keep the indicator, because the 5% who have it may be a specific and interesting group.
• Rows with a missing target must be dropped from training. You cannot impute the answer; doing so trains the model on your imputation rule rather than on reality.
• Rows missing many features can be dropped in training if there are few of them, but note that you will still receive such rows in production and must handle them. Dropping in training and crashing in production is a real deployment pattern.

The rule that keeps you honest

The imputer learns something: a median, a mode, a model of the other columns. It must learn it from training rows only. Compute the median on the whole dataset and the held-out rows have contributed to the value substituted into training, which is leakage: small, invisible, and enough to make your estimate optimistic.

This is not a hypothetical. It is one of the two or three most common ways a validation score comes out better than production, and the fix is to put the imputer inside the pipeline, which the lesson after next covers in full.