Outliers: error, extreme, or the thing you are looking for

Three different things wearing the same label

Before any technique, a classification, because the right action differs completely.

A data error. Age 214. Temperature -300 Celsius. A price of ₹0 on a completed order. These are impossible values and they came from a bug, a default, or a form. Fix or remove them, and record what you did.

A genuine extreme. A real customer who spent ₹40 lakh. Real, rare, and it will happen again. Removing it makes your model wrong about a part of the world that exists.

The signal itself. In fraud detection, intrusion detection or equipment failure, the outliers are the positive class. Removing them removes the entire problem. This is not a hypothetical mistake; it is made regularly by people applying an outlier-removal step from a tutorial.

Most of the damage from outlier handling comes from applying the treatment for the first case to data in the second or third.

Finding them

The two standard rules, both crude, both useful:

IQR rule. Anything below Q1 - 1.5 * IQR or above Q3 + 1.5 * IQR. This is what draws the whiskers on a box plot. It makes no distributional assumption and it is a reasonable first pass.

Z-score. Anything more than 3 standard deviations from the mean. The catch is circular: the standard deviation is itself inflated by the outliers, so a few extreme values raise the threshold and hide themselves. The fix is a modified z-score using the median and the median absolute deviation, which the outliers cannot inflate.

import numpy as np
med = np.median(x)
mad = np.median(np.abs(x - med))
modified_z = 0.6745 * (x - med) / mad

Both of these are one-dimensional. A row can be perfectly ordinary in every column and still be an impossible combination: a 19-year-old with 30 years of work experience passes every column check. Multivariate detection (isolation forests, Mahalanobis distance) is in module 7; for now, knowing that per-column checks miss combinations is enough to stop you trusting them completely.

The four responses

Fix. If you can determine the true value (a decimal point moved, a unit confusion between kg and lbs) fix it. Best outcome, rarely available.

Remove. Only for genuine errors, and only from training. You cannot remove rows from production; whatever you do in training must have a counterpart at serving time.

Clip (winsorise). Cap values at a percentile, typically the 1st and 99th.

lo, hi = np.percentile(x_train, [1, 99])
x = np.clip(x, lo, hi)

This keeps the row and its information while limiting how much any single value can move the fit. It is the pragmatic default for a numeric feature with a heavy tail feeding a linear model. Compute the percentiles on training data only and store them: this is a learned parameter like any other.

Do nothing, but choose a robust method. Often the best answer. Use MAE or Huber loss instead of MSE. Use RobustScaler. Use trees, which are almost entirely insensitive to the magnitude of an extreme value because they only care which side of a threshold it falls on.

Which models care

• Linear regression with MSE: very sensitive. One outlier can visibly tilt the line, as we saw with five houses in module 2.
• Trees and forests: barely care. An extreme value goes into the rightmost leaf and stays there.
• k-NN and k-means: sensitive, because both are built on distances and one distant point distorts them.
• Neural networks: sensitive, and an extreme input can produce a huge gradient that damages the weights in a single step. Gradient clipping exists partly for this reason.

So “should I remove outliers” has no general answer. It depends on the model you are about to fit, which is why the question belongs after the modelling decision rather than before it.

The professional habit

When you remove or clip anything, count it and write the count down. “Removed 412 rows of 1.2 million where age was above 120 or below 0: 0.03%.” That sentence takes ten seconds and does three things: it lets someone else reproduce your dataset, it makes it obvious if you ever remove 15% of the data by accident, and it stops the quiet process by which an analysis becomes cleaner and cleaner until it no longer describes anything real.