{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"missing-values","title":"Missing values, and the information in the gap","url":"https://addaly.com/learn/machine-learning-foundations/missing-values","position":24,"of":92,"minutes":9,"answer":"Add a missingness indicator before imputing, because the fact that a value is absent is frequently more predictive than the value would have been.","keyIdea":"Add a missingness indicator before imputing, because the fact that a value is absent is frequently more predictive than the value would have been, and let boosted trees learn their own direction for NaN rather than imputing at all."},"body":{"format":"markdown","text":"## Why it is missing changes what to do\n\nThree situations, with consequences that differ enough to be worth the vocabulary.\n\n**Missing completely at random.** A sensor dropped packets. Nothing about the row determines whether the value is there. Rare in practice, and the only case where dropping rows is safe.\n\n**Missing at random, given what you observed.** Income is missing more often for younger applicants, but among applicants of the same age, whether income is recorded has nothing to do with the income itself. Imputation using the other columns is principled here.\n\n**Missing not at random.** The value is missing *because of what it is*. People with very high or very low incomes decline to state them. Patients who felt fine did not return for the follow-up test. Here no imputation can recover the truth, because the information needed to reconstruct it is exactly what is absent.\n\nThe uncomfortable part: you cannot distinguish these from the data. It is a judgement about the process that produced the rows, and it comes from talking to whoever collected them.\n\n> Figure: Three reasons a value is missing. Layers, top to bottom: Missing completely at random (A sensor dropped packets. Nothing about the row decides it. Dropping rows is safe here, and only here. Rare.); Missing at random, given what you saw (Younger applicants skip the income field, but among applicants of one age the gap says nothing about income. Impute from the other columns.); Missing not at random (Very high and very low incomes go unstated. The information needed to fill the gap is exactly what is absent. No imputation recovers it; the indicator carries the signal.). The data cannot tell you which of these you have; whoever collected the rows can. In every case the first move is the same: add the indicator column, then impute, or let boosted trees learn a direction for NaN themselves.\n\n## The gap is often the best feature\n\nThe practical response to all three cases starts the same way: **add an indicator column** saying whether the value was missing, then impute.\n\n```python\nfrom sklearn.impute import SimpleImputer\nSimpleImputer(strategy=\"median\", add_indicator=True)\n```\n\nThat one argument is the highest-value thing in this lesson. In real datasets, missingness is frequently more predictive than the value would have been. A loan applicant who left employer blank, a patient with no recorded blood pressure, a form where the optional field is empty: each carries information about the person and the process, and imputing without the flag throws it away permanently.\n\nWorth being clear-eyed about what that means: your model may be learning from an administrative artefact rather than anything about the world. That can be fine, or it can be a fairness problem if missingness correlates with a protected group, or it can break the moment a form changes. Know which you have.\n\n## Choosing an imputation strategy\n\n- **Median** for numeric. Robust to skew, and most real numeric columns are skewed. Mean is fine for symmetric data and dragged around by outliers otherwise.\n- **Most frequent** or an explicit `\"missing\"` category for categoricals. The explicit category is usually better: it lets the model treat missingness as its own level, which is the categorical version of the indicator column.\n- **Constant, out of range** (a value like -999) works for tree models specifically, because a tree can isolate it with one split. It is nonsense for linear models, which will treat -999 as an enormous negative quantity.\n- **KNNImputer** fills from the k most similar rows. More accurate, considerably slower, and it needs scaled features to compute similarity sensibly.\n- **IterativeImputer** models each column from the others in turn. Powerful and slow, and easy to over-trust: a well-imputed value is still a guess presented with the same confidence as a measurement.\n\n## Let the model handle it, when it can\n\nSome models handle missingness natively, and better than imputation:\n\n```python\nfrom sklearn.ensemble import HistGradientBoostingClassifier\nHistGradientBoostingClassifier()   # accepts NaN directly\n```\n\nAt each split, it sends missing values whichever way reduces the loss more, learned separately per split. XGBoost and LightGBM do the same. This is better than imputing, because the model gets to decide what missing means *in each context* rather than having one substituted value everywhere.\n\nIf you are using boosted trees, the correct answer to missing values is often to do nothing.\n\n## When to drop\n\n- **A column that is 95% missing** is usually not worth keeping, although keep the indicator, because the 5% who have it may be a specific and interesting group.\n- **Rows with a missing target** must be dropped from training. You cannot impute the answer; doing so trains the model on your imputation rule rather than on reality.\n- **Rows missing many features** can be dropped in training if there are few of them, but note that you will still receive such rows in production and must handle them. Dropping in training and crashing in production is a real deployment pattern.\n\n## The rule that keeps you honest\n\nThe imputer learns something: a median, a mode, a model of the other columns. It must learn it from **training rows only**. Compute the median on the whole dataset and the held-out rows have contributed to the value substituted into training, which is leakage: small, invisible, and enough to make your estimate optimistic.\n\nThis is not a hypothetical. It is one of the two or three most common ways a validation score comes out better than production, and the fix is to put the imputer inside the pipeline, which the lesson after next covers in full."},"check":{"question":"A hospital dataset has 'last cholesterol reading' missing for 40% of patients, mostly those who never had symptoms. The team imputes with the median and drops the indicator. What have they lost?","options":[{"text":"Nothing important, since median imputation preserves the distribution's centre and the model can still use the other features.","correct":false,"tempting":"Preserving the centre is a property of median imputation, but the concern is the information carried by which patients are missing, not the shape of the imputed column."},{"text":"The signal that the test was never ordered, which encodes a clinician's judgement that the patient was low risk, is now indistinguishable from a genuinely median result.","correct":true},{"text":"The variance of the cholesterol column, which is now understated because 40% of values are identical.","correct":false,"tempting":"Variance shrinkage is a real side effect of single imputation, but it matters far less here than the erasure of a strong risk signal."},{"text":"The ability to use the column at all, since 40% missingness exceeds the usual threshold for retaining a feature.","correct":false,"tempting":"Applies a threshold rule that does not exist; a 40%-missing column with an informative missingness pattern is often among the most useful in the table."}]},"prev":{"slug":"when-a-column-has-thousands-of-levels","title":"When a column has thousands of levels","url":"https://addaly.com/learn/machine-learning-foundations/when-a-column-has-thousands-of-levels"},"next":{"slug":"outliers-and-what-to-do-with-them","title":"Outliers: error, extreme, or the thing you are looking for","url":"https://addaly.com/learn/machine-learning-foundations/outliers-and-what-to-do-with-them"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Missing values, and the information in the gap', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/missing-values (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Missing values, and the information in the gap","courseTitle":"Machine Learning, Foundations","module":{"index":3,"slug":"the-data-is-the-job","title":"The data is the job"},"answer":null,"keyIdea":"Add a missingness indicator before imputing, because the fact that a value is absent is frequently more predictive than the value would have been, and let boosted trees learn their own direction for NaN rather than imputing at all.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"Three reasons a value is missing","layers":[{"label":"Missing completely at random","detail":"A sensor dropped packets. Nothing about the row decides it. Dropping rows is safe here, and only here. Rare."},{"label":"Missing at random, given what you saw","detail":"Younger applicants skip the income field, but among applicants of one age the gap says nothing about income. Impute from the other columns."},{"label":"Missing not at random","detail":"Very high and very low incomes go unstated. The information needed to fill the gap is exactly what is absent. No imputation recovers it; the indicator carries the signal."}],"caption":"The data cannot tell you which of these you have; whoever collected the rows can. In every case the first move is the same: add the indicator column, then impute, or let boosted trees learn a direction for NaN themselves."},"alt":"Figure: Three reasons a value is missing. Layers, top to bottom: Missing completely at random (A sensor dropped packets. Nothing about the row decides it. Dropping rows is safe here, and only here. Rare.); Missing at random, given what you saw (Younger applicants skip the income field, but among applicants of one age the gap says nothing about income. Impute from the other columns.); Missing not at random (Very high and very low incomes go unstated. The information needed to fill the gap is exactly what is absent. No imputation recovers it; the indicator carries the signal.). The data cannot tell you which of these you have; whoever collected the rows can. In every case the first move is the same: add the indicator column, then impute, or let boosted trees learn a direction for NaN themselves."}},"blocks":[{"t":"h2","id":"why-it-is-missing-changes-what-to-do","runs":[{"text":"Why it is missing changes what to do"}]},{"t":"p","runs":[{"text":"Three situations, with consequences that differ enough to be worth the vocabulary."}]},{"t":"p","runs":[{"text":"Missing completely at random.","b":true},{"text":" A sensor dropped packets. Nothing about the row determines whether the value is there. Rare in practice, and the only case where dropping rows is safe."}]},{"t":"p","runs":[{"text":"Missing at random, given what you observed.","b":true},{"text":" Income is missing more often for younger applicants, but among applicants of the same age, whether income is recorded has nothing to do with the income itself. Imputation using the other columns is principled here."}]},{"t":"p","runs":[{"text":"Missing not at random.","b":true},{"text":" The value is missing "},{"text":"because of what it is","i":true},{"text":". People with very high or very low incomes decline to state them. Patients who felt fine did not return for the follow-up test. Here no imputation can recover the truth, because the information needed to reconstruct it is exactly what is absent."}]},{"t":"p","runs":[{"text":"The uncomfortable part: you cannot distinguish these from the data. It is a judgement about the process that produced the rows, and it comes from talking to whoever collected them."}]},{"t":"h2","id":"the-gap-is-often-the-best-feature","runs":[{"text":"The gap is often the best feature"}]},{"t":"p","runs":[{"text":"The practical response to all three cases starts the same way: "},{"text":"add an indicator column","b":true},{"text":" saying whether the value was missing, then impute."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.impute import SimpleImputer\nSimpleImputer(strategy=\"median\", add_indicator=True)"},{"t":"p","runs":[{"text":"That one argument is the highest-value thing in this lesson. In real datasets, missingness is frequently more predictive than the value would have been. A loan applicant who left employer blank, a patient with no recorded blood pressure, a form where the optional field is empty: each carries information about the person and the process, and imputing without the flag throws it away permanently."}]},{"t":"p","runs":[{"text":"Worth being clear-eyed about what that means: your model may be learning from an administrative artefact rather than anything about the world. That can be fine, or it can be a fairness problem if missingness correlates with a protected group, or it can break the moment a form changes. Know which you have."}]},{"t":"h2","id":"choosing-an-imputation-strategy","runs":[{"text":"Choosing an imputation strategy"}]},{"t":"ul","items":[[{"text":"Median","b":true},{"text":" for numeric. Robust to skew, and most real numeric columns are skewed. Mean is fine for symmetric data and dragged around by outliers otherwise."}],[{"text":"Most frequent","b":true},{"text":" or an explicit "},{"text":"\"missing\"","code":true},{"text":" category for categoricals. The explicit category is usually better: it lets the model treat missingness as its own level, which is the categorical version of the indicator column."}],[{"text":"Constant, out of range","b":true},{"text":" (a value like -999) works for tree models specifically, because a tree can isolate it with one split. It is nonsense for linear models, which will treat -999 as an enormous negative quantity."}],[{"text":"KNNImputer","b":true},{"text":" fills from the k most similar rows. More accurate, considerably slower, and it needs scaled features to compute similarity sensibly."}],[{"text":"IterativeImputer","b":true},{"text":" models each column from the others in turn. Powerful and slow, and easy to over-trust: a well-imputed value is still a guess presented with the same confidence as a measurement."}]]},{"t":"h2","id":"let-the-model-handle-it-when-it-can","runs":[{"text":"Let the model handle it, when it can"}]},{"t":"p","runs":[{"text":"Some models handle missingness natively, and better than imputation:"}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.ensemble import HistGradientBoostingClassifier\nHistGradientBoostingClassifier()   # accepts NaN directly"},{"t":"p","runs":[{"text":"At each split, it sends missing values whichever way reduces the loss more, learned separately per split. XGBoost and LightGBM do the same. This is better than imputing, because the model gets to decide what missing means "},{"text":"in each context","i":true},{"text":" rather than having one substituted value everywhere."}]},{"t":"p","runs":[{"text":"If you are using boosted trees, the correct answer to missing values is often to do nothing."}]},{"t":"h2","id":"when-to-drop","runs":[{"text":"When to drop"}]},{"t":"ul","items":[[{"text":"A column that is 95% missing","b":true},{"text":" is usually not worth keeping, although keep the indicator, because the 5% who have it may be a specific and interesting group."}],[{"text":"Rows with a missing target","b":true},{"text":" must be dropped from training. You cannot impute the answer; doing so trains the model on your imputation rule rather than on reality."}],[{"text":"Rows missing many features","b":true},{"text":" can be dropped in training if there are few of them, but note that you will still receive such rows in production and must handle them. Dropping in training and crashing in production is a real deployment pattern."}]]},{"t":"h2","id":"the-rule-that-keeps-you-honest","runs":[{"text":"The rule that keeps you honest"}]},{"t":"p","runs":[{"text":"The imputer learns something: a median, a mode, a model of the other columns. It must learn it from "},{"text":"training rows only","b":true},{"text":". Compute the median on the whole dataset and the held-out rows have contributed to the value substituted into training, which is leakage: small, invisible, and enough to make your estimate optimistic."}]},{"t":"p","runs":[{"text":"This is not a hypothetical. It is one of the two or three most common ways a validation score comes out better than production, and the fix is to put the imputer inside the pipeline, which the lesson after next covers in full."}]}],"check":{"question":[{"text":"A hospital dataset has ‘last cholesterol reading’ missing for 40% of patients, mostly those who never had symptoms. The team imputes with the median and drops the indicator. What have they lost?"}],"options":[{"text":[{"text":"Nothing important, since median imputation preserves the distribution’s centre and the model can still use the other features."}],"correct":false,"tempting":[{"text":"Preserving the centre is a property of median imputation, but the concern is the information carried by which patients are missing, not the shape of the imputed column."}]},{"text":[{"text":"The signal that the test was never ordered, which encodes a clinician’s judgement that the patient was low risk, is now indistinguishable from a genuinely median result."}],"correct":true,"tempting":null},{"text":[{"text":"The variance of the cholesterol column, which is now understated because 40% of values are identical."}],"correct":false,"tempting":[{"text":"Variance shrinkage is a real side effect of single imputation, but it matters far less here than the erasure of a strong risk signal."}]},{"text":[{"text":"The ability to use the column at all, since 40% missingness exceeds the usual threshold for retaining a feature."}],"correct":false,"tempting":[{"text":"Applies a threshold rule that does not exist; a 40%-missing column with an informative missingness pattern is often among the most useful in the table."}]}],"order":[0,2,3,1],"why":null},"prev":{"slug":"when-a-column-has-thousands-of-levels","title":"When a column has thousands of levels","minutes":8},"next":{"slug":"outliers-and-what-to-do-with-them","title":"Outliers: error, extreme, or the thing you are looking for","minutes":8}}}