{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"outliers-and-what-to-do-with-them","title":"Outliers: error, extreme, or the thing you are looking for","url":"https://addaly.com/learn/machine-learning-foundations/outliers-and-what-to-do-with-them","position":25,"of":92,"minutes":8,"answer":"Decide first whether an extreme value is an error, a genuine rarity, or the target itself, because the same removal step is correct in the first case, harmful in the second.","keyIdea":"Decide first whether an extreme value is an error, a genuine rarity, or the target itself, because the same removal step is correct in the first case, harmful in the second, and destroys the problem in the third."},"body":{"format":"markdown","text":"## Three different things wearing the same label\n\nBefore any technique, a classification, because the right action differs completely.\n\n**A data error.** Age 214. Temperature -300 Celsius. A price of ₹0 on a completed order. These are impossible values and they came from a bug, a default, or a form. Fix or remove them, and record what you did.\n\n**A genuine extreme.** A real customer who spent ₹40 lakh. Real, rare, and it will happen again. Removing it makes your model wrong about a part of the world that exists.\n\n**The signal itself.** In fraud detection, intrusion detection or equipment failure, the outliers are the positive class. Removing them removes the entire problem. This is not a hypothetical mistake; it is made regularly by people applying an outlier-removal step from a tutorial.\n\nMost of the damage from outlier handling comes from applying the treatment for the first case to data in the second or third.\n\n## Finding them\n\nThe two standard rules, both crude, both useful:\n\n**IQR rule.** Anything below `Q1 - 1.5 * IQR` or above `Q3 + 1.5 * IQR`. This is what draws the whiskers on a box plot. It makes no distributional assumption and it is a reasonable first pass.\n\n**Z-score.** Anything more than 3 standard deviations from the mean. The catch is circular: the standard deviation is itself inflated by the outliers, so a few extreme values raise the threshold and hide themselves. The fix is a modified z-score using the median and the median absolute deviation, which the outliers cannot inflate.\n\n```python\nimport numpy as np\nmed = np.median(x)\nmad = np.median(np.abs(x - med))\nmodified_z = 0.6745 * (x - med) / mad\n```\n\nBoth of these are one-dimensional. A row can be perfectly ordinary in every column and still be an impossible combination: a 19-year-old with 30 years of work experience passes every column check. Multivariate detection (isolation forests, Mahalanobis distance) is in module 7; for now, knowing that per-column checks miss combinations is enough to stop you trusting them completely.\n\n## The four responses\n\n**Fix.** If you can determine the true value (a decimal point moved, a unit confusion between kg and lbs) fix it. Best outcome, rarely available.\n\n**Remove.** Only for genuine errors, and only from training. You cannot remove rows from production; whatever you do in training must have a counterpart at serving time.\n\n**Clip (winsorise).** Cap values at a percentile, typically the 1st and 99th.\n\n```python\nlo, hi = np.percentile(x_train, [1, 99])\nx = np.clip(x, lo, hi)\n```\n\nThis keeps the row and its information while limiting how much any single value can move the fit. It is the pragmatic default for a numeric feature with a heavy tail feeding a linear model. Compute the percentiles on training data only and store them: this is a learned parameter like any other.\n\n**Do nothing, but choose a robust method.** Often the best answer. Use MAE or Huber loss instead of MSE. Use `RobustScaler`. Use trees, which are almost entirely insensitive to the magnitude of an extreme value because they only care which side of a threshold it falls on.\n\n## Which models care\n\n- **Linear regression with MSE**: very sensitive. One outlier can visibly tilt the line, as we saw with five houses in module 2.\n- **Trees and forests**: barely care. An extreme value goes into the rightmost leaf and stays there.\n- **k-NN and k-means**: sensitive, because both are built on distances and one distant point distorts them.\n- **Neural networks**: sensitive, and an extreme input can produce a huge gradient that damages the weights in a single step. Gradient clipping exists partly for this reason.\n\nSo \"should I remove outliers\" has no general answer. It depends on the model you are about to fit, which is why the question belongs after the modelling decision rather than before it.\n\n## The professional habit\n\nWhen you remove or clip anything, count it and write the count down. \"Removed 412 rows of 1.2 million where age was above 120 or below 0: 0.03%.\" That sentence takes ten seconds and does three things: it lets someone else reproduce your dataset, it makes it obvious if you ever remove 15% of the data by accident, and it stops the quiet process by which an analysis becomes cleaner and cleaner until it no longer describes anything real."},"check":{"question":"An analyst applies a 3-standard-deviation filter to transaction amounts before training a fraud detector, removing 0.4% of rows. Recall on fraud collapses. Why?","options":[{"text":"Removing 0.4% of rows shrank the training set enough to increase variance in the estimated model.","correct":false,"tempting":"A 0.4% reduction is far too small to matter for variance, and would not cause a collapse in one specific class's recall."},{"text":"Fraudulent transactions are disproportionately extreme in amount, so the filter preferentially deleted the positive class the model needed to learn.","correct":true},{"text":"The z-score threshold was computed after scaling, so it removed the wrong rows.","correct":false,"tempting":"Order of scaling and thresholding changes which rows are flagged only if the scaler is non-monotonic; standard scaling preserves the same set of extreme rows."},{"text":"Removing rows changed the class balance, so the decision threshold of 0.5 became miscalibrated.","correct":false,"tempting":"Class balance did shift slightly, but a 0.4% removal cannot move the base rate enough to explain a collapse, and rebalancing would not restore deleted examples."}]},"prev":{"slug":"missing-values","title":"Missing values, and the information in the gap","url":"https://addaly.com/learn/machine-learning-foundations/missing-values"},"next":{"slug":"dates-and-cycles","title":"Dates, durations, and the trouble with midnight","url":"https://addaly.com/learn/machine-learning-foundations/dates-and-cycles"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Outliers: error, extreme, or the thing you are looking for', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/outliers-and-what-to-do-with-them (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Outliers: error, extreme, or the thing you are looking for","courseTitle":"Machine Learning, Foundations","module":{"index":3,"slug":"the-data-is-the-job","title":"The data is the job"},"answer":null,"keyIdea":"Decide first whether an extreme value is an error, a genuine rarity, or the target itself, because the same removal step is correct in the first case, harmful in the second, and destroys the problem in the third.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"three-different-things-wearing-the-same-label","runs":[{"text":"Three different things wearing the same label"}]},{"t":"p","runs":[{"text":"Before any technique, a classification, because the right action differs completely."}]},{"t":"p","runs":[{"text":"A data error.","b":true},{"text":" Age 214. Temperature -300 Celsius. A price of ₹0 on a completed order. These are impossible values and they came from a bug, a default, or a form. Fix or remove them, and record what you did."}]},{"t":"p","runs":[{"text":"A genuine extreme.","b":true},{"text":" A real customer who spent ₹40 lakh. Real, rare, and it will happen again. Removing it makes your model wrong about a part of the world that exists."}]},{"t":"p","runs":[{"text":"The signal itself.","b":true},{"text":" In fraud detection, intrusion detection or equipment failure, the outliers are the positive class. Removing them removes the entire problem. This is not a hypothetical mistake; it is made regularly by people applying an outlier-removal step from a tutorial."}]},{"t":"p","runs":[{"text":"Most of the damage from outlier handling comes from applying the treatment for the first case to data in the second or third."}]},{"t":"h2","id":"finding-them","runs":[{"text":"Finding them"}]},{"t":"p","runs":[{"text":"The two standard rules, both crude, both useful:"}]},{"t":"p","runs":[{"text":"IQR rule.","b":true},{"text":" Anything below "},{"text":"Q1 - 1.5 * IQR","code":true},{"text":" or above "},{"text":"Q3 + 1.5 * IQR","code":true},{"text":". This is what draws the whiskers on a box plot. It makes no distributional assumption and it is a reasonable first pass."}]},{"t":"p","runs":[{"text":"Z-score.","b":true},{"text":" Anything more than 3 standard deviations from the mean. The catch is circular: the standard deviation is itself inflated by the outliers, so a few extreme values raise the threshold and hide themselves. The fix is a modified z-score using the median and the median absolute deviation, which the outliers cannot inflate."}]},{"t":"code","lang":"python","label":"Python","text":"import numpy as np\nmed = np.median(x)\nmad = np.median(np.abs(x - med))\nmodified_z = 0.6745 * (x - med) / mad"},{"t":"p","runs":[{"text":"Both of these are one-dimensional. A row can be perfectly ordinary in every column and still be an impossible combination: a 19-year-old with 30 years of work experience passes every column check. Multivariate detection (isolation forests, Mahalanobis distance) is in module 7; for now, knowing that per-column checks miss combinations is enough to stop you trusting them completely."}]},{"t":"h2","id":"the-four-responses","runs":[{"text":"The four responses"}]},{"t":"p","runs":[{"text":"Fix.","b":true},{"text":" If you can determine the true value (a decimal point moved, a unit confusion between kg and lbs) fix it. Best outcome, rarely available."}]},{"t":"p","runs":[{"text":"Remove.","b":true},{"text":" Only for genuine errors, and only from training. You cannot remove rows from production; whatever you do in training must have a counterpart at serving time."}]},{"t":"p","runs":[{"text":"Clip (winsorise).","b":true},{"text":" Cap values at a percentile, typically the 1st and 99th."}]},{"t":"code","lang":"python","label":"Python","text":"lo, hi = np.percentile(x_train, [1, 99])\nx = np.clip(x, lo, hi)"},{"t":"p","runs":[{"text":"This keeps the row and its information while limiting how much any single value can move the fit. It is the pragmatic default for a numeric feature with a heavy tail feeding a linear model. Compute the percentiles on training data only and store them: this is a learned parameter like any other."}]},{"t":"p","runs":[{"text":"Do nothing, but choose a robust method.","b":true},{"text":" Often the best answer. Use MAE or Huber loss instead of MSE. Use "},{"text":"RobustScaler","code":true},{"text":". Use trees, which are almost entirely insensitive to the magnitude of an extreme value because they only care which side of a threshold it falls on."}]},{"t":"h2","id":"which-models-care","runs":[{"text":"Which models care"}]},{"t":"ul","items":[[{"text":"Linear regression with MSE","b":true},{"text":": very sensitive. One outlier can visibly tilt the line, as we saw with five houses in module 2."}],[{"text":"Trees and forests","b":true},{"text":": barely care. An extreme value goes into the rightmost leaf and stays there."}],[{"text":"k-NN and k-means","b":true},{"text":": sensitive, because both are built on distances and one distant point distorts them."}],[{"text":"Neural networks","b":true},{"text":": sensitive, and an extreme input can produce a huge gradient that damages the weights in a single step. Gradient clipping exists partly for this reason."}]]},{"t":"p","runs":[{"text":"So “should I remove outliers” has no general answer. It depends on the model you are about to fit, which is why the question belongs after the modelling decision rather than before it."}]},{"t":"h2","id":"the-professional-habit","runs":[{"text":"The professional habit"}]},{"t":"p","runs":[{"text":"When you remove or clip anything, count it and write the count down. “Removed 412 rows of 1.2 million where age was above 120 or below 0: 0.03%.” That sentence takes ten seconds and does three things: it lets someone else reproduce your dataset, it makes it obvious if you ever remove 15% of the data by accident, and it stops the quiet process by which an analysis becomes cleaner and cleaner until it no longer describes anything real."}]}],"check":{"question":[{"text":"An analyst applies a 3-standard-deviation filter to transaction amounts before training a fraud detector, removing 0.4% of rows. Recall on fraud collapses. Why?"}],"options":[{"text":[{"text":"Removing 0.4% of rows shrank the training set enough to increase variance in the estimated model."}],"correct":false,"tempting":[{"text":"A 0.4% reduction is far too small to matter for variance, and would not cause a collapse in one specific class’s recall."}]},{"text":[{"text":"Fraudulent transactions are disproportionately extreme in amount, so the filter preferentially deleted the positive class the model needed to learn."}],"correct":true,"tempting":null},{"text":[{"text":"The z-score threshold was computed after scaling, so it removed the wrong rows."}],"correct":false,"tempting":[{"text":"Order of scaling and thresholding changes which rows are flagged only if the scaler is non-monotonic; standard scaling preserves the same set of extreme rows."}]},{"text":[{"text":"Removing rows changed the class balance, so the decision threshold of 0.5 became miscalibrated."}],"correct":false,"tempting":[{"text":"Class balance did shift slightly, but a 0.4% removal cannot move the base rate enough to explain a collapse, and rebalancing would not restore deleted examples."}]}],"order":[2,0,3,1],"why":null},"prev":{"slug":"missing-values","title":"Missing values, and the information in the gap","minutes":9},"next":{"slug":"dates-and-cycles","title":"Dates, durations, and the trouble with midnight","minutes":8}}}