{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"numerical-features","title":"Scaling, and the transforms that change what a model can see","url":"https://addaly.com/learn/machine-learning-foundations/numerical-features","position":21,"of":92,"minutes":9,"answer":"Scaling is required by distance-based, penalised and gradient-fitted models and irrelevant to trees; the log transform is not cosmetic but a claim that ratios matter more than differences.","keyIdea":"Scaling is required by distance-based, penalised and gradient-fitted models and irrelevant to trees; the log transform is not cosmetic but a claim that ratios matter more than differences."},"body":{"format":"markdown","text":"## Which models need scaling, and why\n\nScaling is not a universal good. It is required by a specific set of models for a specific reason, and applying it blindly wastes time while not applying it where needed silently ruins results.\n\n**Needs scaling:**\n\n- Anything with a **distance** in it: k-nearest neighbours, k-means, SVMs with RBF kernels. If income runs to millions and age to tens, the distance between two people is essentially the difference in their incomes; age contributes nothing.\n- Anything with a **penalty on coefficients**: ridge, lasso, regularised logistic regression. As module 2 showed, the penalty applies to coefficient size, so it lands on features by unit of measurement rather than by importance.\n- Anything fitted by **gradient descent**: neural networks, `SGDClassifier`. Mismatched scales create a long, narrow loss surface that one learning rate cannot navigate.\n- **PCA**, because it maximises variance, and a feature in rupees has more variance than one in kilometres for no meaningful reason.\n\n**Does not need scaling:** decision trees, random forests, and every gradient boosting library. A tree asks \"is `income > 450000`\", and the answer does not change if you divide the whole column by a thousand. Scaling before XGBoost is harmless and pointless.\n\n## The three scalers, and when each is right\n\n```python\nfrom sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler\n```\n\n**StandardScaler** subtracts the mean and divides by the standard deviation. Result has mean 0 and standard deviation 1. The default; use it unless you have a reason not to.\n\n**MinMaxScaler** maps the observed range to [0, 1]. Useful when a bounded range is required: some neural network inputs, image pixels. Fragile against outliers: one value of 10 million squashes everything else into the bottom 0.1% of the range.\n\n**RobustScaler** subtracts the median and divides by the interquartile range. Because the median and IQR ignore the tails, extreme values do not move the scaling of the bulk. This is the right default when you know the data has heavy tails and you do not want to remove them.\n\n## Log, and what it actually does\n\nMany real quantities are multiplicative rather than additive: income, city population, page views, transaction sizes, time between events. They are strongly right-skewed, with most values small and a long tail of very large ones.\n\n```python\nimport numpy as np\nX[\"log_income\"] = np.log1p(X[\"income\"])\n```\n\nUse `log1p`, which computes `log(1 + x)` and therefore handles zeros. On negative values the log is undefined and you need a different tool.\n\nWhat the log changes conceptually is which differences count as equal. On the raw scale, 10,000 to 20,000 and 1,000,000 to 1,010,000 are the same difference. On the log scale, 10,000 to 20,000 and 1,000,000 to 2,000,000 are the same difference. The second framing is usually closer to how the world works: a doubling of income means something similar wherever it starts, while an extra ₹10,000 does not.\n\nThat is not a trick to make a histogram prettier. It is a claim about the domain, and it is often the single highest-value transformation you can apply to a linear model.\n\n## When you cannot guess the right transform\n\n**QuantileTransformer** maps a feature to a uniform or normal distribution by rank. It handles any shape, ignores outliers by construction, and is a reasonable choice when you have no theory about the feature.\n\nIts cost is real: it destroys the original units, so coefficients become uninterpretable, and it can turn a meaningful gap into an ordinary one. A feature where 99% of rows are 0 and 1% are enormous will be flattened into something a model can use but nobody can explain.\n\n**PowerTransformer** with the Yeo-Johnson method finds a power transformation that makes the distribution as symmetric as possible, and it works with zeros and negatives. It is a good middle option: milder than quantile, more flexible than a fixed log.\n\n## Ratios and differences beat raw columns\n\nThe transformation that most often wins is not on the list above, because it uses two columns.\n\n- `debt` and `income` separately are weaker than `debt / income`. The ratio is what a credit officer actually looks at.\n- `total_spend` and `n_orders` separately are weaker than `average_order_value`.\n- `date_of_payment` and `date_due` are weaker than `days_late`.\n\nA tree can approximate a ratio with a staircase of splits, but it takes many splits and a lot of data to do badly what one division does exactly. A linear model cannot do it at all without the column. Sit with the domain for twenty minutes and write down the four or five quantities a knowledgeable person computes in their head; those are usually your best features and they are essentially free.\n\n## Do it inside the pipeline\n\nThe scaler learns parameters: a mean, a standard deviation, quantile boundaries. Those must be learned from training rows only and then applied unchanged to validation, test, and production data. Fit a `StandardScaler` on the full dataset and you have moved information from the held-out rows into the training process, which inflates your score by a small amount you cannot measure. Two lessons on, we will make that mechanism explicit."},"check":{"question":"A team standardises every numeric column before training an XGBoost model and reports no change in accuracy. What does this tell them?","options":[{"text":"The scaler was fitted on the wrong subset, so the transformation had no effect.","correct":false,"tempting":"Assumes a bug, but a correctly fitted scaler on tree input also produces no accuracy change, so the null result is expected rather than diagnostic."},{"text":"The features were already close to standard normal, so the transformation was near-identity.","correct":false,"tempting":"Would explain no change for a scale-sensitive model, but the result holds for trees regardless of the original distribution."},{"text":"Boosted trees split on thresholds within a single feature, so any monotonic rescaling produces the same splits and the same predictions.","correct":true},{"text":"XGBoost internally standardises its inputs, making external scaling redundant.","correct":false,"tempting":"Invents an internal preprocessing step; XGBoost does no such thing, and it does not need to because splits are scale-invariant."}]},"prev":{"slug":"features-beat-models","title":"Features beat models","url":"https://addaly.com/learn/machine-learning-foundations/features-beat-models"},"next":{"slug":"categorical-encoding","title":"Turning categories into numbers without lying about them","url":"https://addaly.com/learn/machine-learning-foundations/categorical-encoding"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Scaling, and the transforms that change what a model can see', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/numerical-features (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Scaling, and the transforms that change what a model can see","courseTitle":"Machine Learning, Foundations","module":{"index":3,"slug":"the-data-is-the-job","title":"The data is the job"},"answer":null,"keyIdea":"Scaling is required by distance-based, penalised and gradient-fitted models and irrelevant to trees; the log transform is not cosmetic but a claim that ratios matter more than differences.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"which-models-need-scaling-and-why","runs":[{"text":"Which models need scaling, and why"}]},{"t":"p","runs":[{"text":"Scaling is not a universal good. It is required by a specific set of models for a specific reason, and applying it blindly wastes time while not applying it where needed silently ruins results."}]},{"t":"p","runs":[{"text":"Needs scaling:","b":true}]},{"t":"ul","items":[[{"text":"Anything with a "},{"text":"distance","b":true},{"text":" in it: k-nearest neighbours, k-means, SVMs with RBF kernels. If income runs to millions and age to tens, the distance between two people is essentially the difference in their incomes; age contributes nothing."}],[{"text":"Anything with a "},{"text":"penalty on coefficients","b":true},{"text":": ridge, lasso, regularised logistic regression. As module 2 showed, the penalty applies to coefficient size, so it lands on features by unit of measurement rather than by importance."}],[{"text":"Anything fitted by "},{"text":"gradient descent","b":true},{"text":": neural networks, "},{"text":"SGDClassifier","code":true},{"text":". Mismatched scales create a long, narrow loss surface that one learning rate cannot navigate."}],[{"text":"PCA","b":true},{"text":", because it maximises variance, and a feature in rupees has more variance than one in kilometres for no meaningful reason."}]]},{"t":"p","runs":[{"text":"Does not need scaling:","b":true},{"text":" decision trees, random forests, and every gradient boosting library. A tree asks “is "},{"text":"income > 450000","code":true},{"text":"”, and the answer does not change if you divide the whole column by a thousand. Scaling before XGBoost is harmless and pointless."}]},{"t":"h2","id":"the-three-scalers-and-when-each-is-right","runs":[{"text":"The three scalers, and when each is right"}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler"},{"t":"p","runs":[{"text":"StandardScaler","b":true},{"text":" subtracts the mean and divides by the standard deviation. Result has mean 0 and standard deviation 1. The default; use it unless you have a reason not to."}]},{"t":"p","runs":[{"text":"MinMaxScaler","b":true},{"text":" maps the observed range to [0, 1]. Useful when a bounded range is required: some neural network inputs, image pixels. Fragile against outliers: one value of 10 million squashes everything else into the bottom 0.1% of the range."}]},{"t":"p","runs":[{"text":"RobustScaler","b":true},{"text":" subtracts the median and divides by the interquartile range. Because the median and IQR ignore the tails, extreme values do not move the scaling of the bulk. This is the right default when you know the data has heavy tails and you do not want to remove them."}]},{"t":"h2","id":"log-and-what-it-actually-does","runs":[{"text":"Log, and what it actually does"}]},{"t":"p","runs":[{"text":"Many real quantities are multiplicative rather than additive: income, city population, page views, transaction sizes, time between events. They are strongly right-skewed, with most values small and a long tail of very large ones."}]},{"t":"code","lang":"python","label":"Python","text":"import numpy as np\nX[\"log_income\"] = np.log1p(X[\"income\"])"},{"t":"p","runs":[{"text":"Use "},{"text":"log1p","code":true},{"text":", which computes "},{"text":"log(1 + x)","code":true},{"text":" and therefore handles zeros. On negative values the log is undefined and you need a different tool."}]},{"t":"p","runs":[{"text":"What the log changes conceptually is which differences count as equal. On the raw scale, 10,000 to 20,000 and 1,000,000 to 1,010,000 are the same difference. On the log scale, 10,000 to 20,000 and 1,000,000 to 2,000,000 are the same difference. The second framing is usually closer to how the world works: a doubling of income means something similar wherever it starts, while an extra ₹10,000 does not."}]},{"t":"p","runs":[{"text":"That is not a trick to make a histogram prettier. It is a claim about the domain, and it is often the single highest-value transformation you can apply to a linear model."}]},{"t":"h2","id":"when-you-cannot-guess-the-right-transform","runs":[{"text":"When you cannot guess the right transform"}]},{"t":"p","runs":[{"text":"QuantileTransformer","b":true},{"text":" maps a feature to a uniform or normal distribution by rank. It handles any shape, ignores outliers by construction, and is a reasonable choice when you have no theory about the feature."}]},{"t":"p","runs":[{"text":"Its cost is real: it destroys the original units, so coefficients become uninterpretable, and it can turn a meaningful gap into an ordinary one. A feature where 99% of rows are 0 and 1% are enormous will be flattened into something a model can use but nobody can explain."}]},{"t":"p","runs":[{"text":"PowerTransformer","b":true},{"text":" with the Yeo-Johnson method finds a power transformation that makes the distribution as symmetric as possible, and it works with zeros and negatives. It is a good middle option: milder than quantile, more flexible than a fixed log."}]},{"t":"h2","id":"ratios-and-differences-beat-raw-columns","runs":[{"text":"Ratios and differences beat raw columns"}]},{"t":"p","runs":[{"text":"The transformation that most often wins is not on the list above, because it uses two columns."}]},{"t":"ul","items":[[{"text":"debt","code":true},{"text":" and "},{"text":"income","code":true},{"text":" separately are weaker than "},{"text":"debt / income","code":true},{"text":". The ratio is what a credit officer actually looks at."}],[{"text":"total_spend","code":true},{"text":" and "},{"text":"n_orders","code":true},{"text":" separately are weaker than "},{"text":"average_order_value","code":true},{"text":"."}],[{"text":"date_of_payment","code":true},{"text":" and "},{"text":"date_due","code":true},{"text":" are weaker than "},{"text":"days_late","code":true},{"text":"."}]]},{"t":"p","runs":[{"text":"A tree can approximate a ratio with a staircase of splits, but it takes many splits and a lot of data to do badly what one division does exactly. A linear model cannot do it at all without the column. Sit with the domain for twenty minutes and write down the four or five quantities a knowledgeable person computes in their head; those are usually your best features and they are essentially free."}]},{"t":"h2","id":"do-it-inside-the-pipeline","runs":[{"text":"Do it inside the pipeline"}]},{"t":"p","runs":[{"text":"The scaler learns parameters: a mean, a standard deviation, quantile boundaries. Those must be learned from training rows only and then applied unchanged to validation, test, and production data. Fit a "},{"text":"StandardScaler","code":true},{"text":" on the full dataset and you have moved information from the held-out rows into the training process, which inflates your score by a small amount you cannot measure. Two lessons on, we will make that mechanism explicit."}]}],"check":{"question":[{"text":"A team standardises every numeric column before training an XGBoost model and reports no change in accuracy. What does this tell them?"}],"options":[{"text":[{"text":"The scaler was fitted on the wrong subset, so the transformation had no effect."}],"correct":false,"tempting":[{"text":"Assumes a bug, but a correctly fitted scaler on tree input also produces no accuracy change, so the null result is expected rather than diagnostic."}]},{"text":[{"text":"The features were already close to standard normal, so the transformation was near-identity."}],"correct":false,"tempting":[{"text":"Would explain no change for a scale-sensitive model, but the result holds for trees regardless of the original distribution."}]},{"text":[{"text":"Boosted trees split on thresholds within a single feature, so any monotonic rescaling produces the same splits and the same predictions."}],"correct":true,"tempting":null},{"text":[{"text":"XGBoost internally standardises its inputs, making external scaling redundant."}],"correct":false,"tempting":[{"text":"Invents an internal preprocessing step; XGBoost does no such thing, and it does not need to because splits are scale-invariant."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"features-beat-models","title":"Features beat models","minutes":9},"next":{"slug":"categorical-encoding","title":"Turning categories into numbers without lying about them","minutes":9}}}