Scaling, and the transforms that change what a model can see

Which models need scaling, and why

Scaling is not a universal good. It is required by a specific set of models for a specific reason, and applying it blindly wastes time while not applying it where needed silently ruins results.

Needs scaling:

• Anything with a distance in it: k-nearest neighbours, k-means, SVMs with RBF kernels. If income runs to millions and age to tens, the distance between two people is essentially the difference in their incomes; age contributes nothing.
• Anything with a penalty on coefficients: ridge, lasso, regularised logistic regression. As module 2 showed, the penalty applies to coefficient size, so it lands on features by unit of measurement rather than by importance.
• Anything fitted by gradient descent: neural networks, SGDClassifier. Mismatched scales create a long, narrow loss surface that one learning rate cannot navigate.
• PCA, because it maximises variance, and a feature in rupees has more variance than one in kilometres for no meaningful reason.

Does not need scaling: decision trees, random forests, and every gradient boosting library. A tree asks “is income > 450000”, and the answer does not change if you divide the whole column by a thousand. Scaling before XGBoost is harmless and pointless.

The three scalers, and when each is right

from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler

StandardScaler subtracts the mean and divides by the standard deviation. Result has mean 0 and standard deviation 1. The default; use it unless you have a reason not to.

MinMaxScaler maps the observed range to [0, 1]. Useful when a bounded range is required: some neural network inputs, image pixels. Fragile against outliers: one value of 10 million squashes everything else into the bottom 0.1% of the range.

RobustScaler subtracts the median and divides by the interquartile range. Because the median and IQR ignore the tails, extreme values do not move the scaling of the bulk. This is the right default when you know the data has heavy tails and you do not want to remove them.

Log, and what it actually does

Many real quantities are multiplicative rather than additive: income, city population, page views, transaction sizes, time between events. They are strongly right-skewed, with most values small and a long tail of very large ones.

import numpy as np
X["log_income"] = np.log1p(X["income"])

Use log1p, which computes log(1 + x) and therefore handles zeros. On negative values the log is undefined and you need a different tool.

What the log changes conceptually is which differences count as equal. On the raw scale, 10,000 to 20,000 and 1,000,000 to 1,010,000 are the same difference. On the log scale, 10,000 to 20,000 and 1,000,000 to 2,000,000 are the same difference. The second framing is usually closer to how the world works: a doubling of income means something similar wherever it starts, while an extra ₹10,000 does not.

That is not a trick to make a histogram prettier. It is a claim about the domain, and it is often the single highest-value transformation you can apply to a linear model.

When you cannot guess the right transform

QuantileTransformer maps a feature to a uniform or normal distribution by rank. It handles any shape, ignores outliers by construction, and is a reasonable choice when you have no theory about the feature.

Its cost is real: it destroys the original units, so coefficients become uninterpretable, and it can turn a meaningful gap into an ordinary one. A feature where 99% of rows are 0 and 1% are enormous will be flattened into something a model can use but nobody can explain.

PowerTransformer with the Yeo-Johnson method finds a power transformation that makes the distribution as symmetric as possible, and it works with zeros and negatives. It is a good middle option: milder than quantile, more flexible than a fixed log.

Ratios and differences beat raw columns

The transformation that most often wins is not on the list above, because it uses two columns.

• debt and income separately are weaker than debt / income. The ratio is what a credit officer actually looks at.
• total_spend and n_orders separately are weaker than average_order_value.
• date_of_payment and date_due are weaker than days_late.

A tree can approximate a ratio with a staircase of splits, but it takes many splits and a lot of data to do badly what one division does exactly. A linear model cannot do it at all without the column. Sit with the domain for twenty minutes and write down the four or five quantities a knowledgeable person computes in their head; those are usually your best features and they are essentially free.

Do it inside the pipeline

The scaler learns parameters: a mean, a standard deviation, quantile boundaries. Those must be learned from training rows only and then applied unchanged to validation, test, and production data. Fit a StandardScaler on the full dataset and you have moved information from the held-out rows into the training process, which inflates your score by a small amount you cannot measure. Two lessons on, we will make that mechanism explicit.