{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"ridge-and-lasso","title":"Regularisation: paying for large coefficients","url":"https://addaly.com/learn/machine-learning-foundations/ridge-and-lasso","position":17,"of":92,"minutes":9,"answer":"Ridge and lasso add a price on coefficient size to the loss; lasso reaches exactly zero because the gradient of the absolute value stays constant as the coefficient shrinks.","keyIdea":"Ridge and lasso add a price on coefficient size to the loss; lasso reaches exactly zero because the gradient of the absolute value stays constant as the coefficient shrinks, while ridge's push fades away near zero."},"body":{"format":"markdown","text":"## The idea in one sentence\n\nAdd the size of the coefficients to the loss, so the optimiser has to justify every unit of coefficient it spends with a matching reduction in error.\n\n```text\nordinary:  minimise  sum of squared errors\nridge:     minimise  sum of squared errors + alpha * sum of w^2\nlasso:     minimise  sum of squared errors + alpha * sum of |w|\n```\n\nThat is the whole mechanism. `alpha` is the price. At `alpha = 0` you are back to ordinary least squares. As `alpha` rises, coefficients shrink towards zero and the model gets simpler whether it likes it or not.\n\n## Why shrinking helps\n\nA model overfits by using large, opposing coefficients to thread through individual training points. Two correlated features with `+900` and `-880` can produce wild predictions from small input changes: a tiny shift in either feature moves the output enormously. That is a model with high variance: refit it on a different sample and it makes different predictions.\n\nMaking large coefficients expensive removes that option. The model must explain the data with modest weights or not at all. You give up a little training accuracy and buy a large reduction in how much the model changes between samples. That trade is worthwhile whenever you have many features relative to rows, and it is why ridge is a sensible default rather than a rescue.\n\n## The difference between L2 and L1, mechanically\n\nEveryone learns that lasso sets coefficients to exactly zero and ridge does not. Fewer learn why, and the reason is simple enough to hold.\n\nConsider the penalty's own gradient: the push it exerts on a coefficient.\n\n- Ridge penalises `w^2`. Its gradient is `2w`. As `w` approaches zero, the push approaches zero. The penalty loses interest in small coefficients, so they shrink and shrink and never quite arrive.\n- Lasso penalises `|w|`. Its gradient is a constant (the sign of `w`) regardless of size. A coefficient of 0.001 gets exactly the same push toward zero as a coefficient of 50. That constant push is strong enough to drive small coefficients to exactly zero and hold them there.\n\nSo lasso performs feature selection as a side effect of its penalty shape, not as a separate step. With `alpha` tuned, a lasso on 200 features may keep 23 and zero the rest.\n\n> Figure: The push each penalty exerts on a coefficient. Line chart of Push towards zero against Coefficient size, w: Lasso: gradient of |w| is constant (0.01, 1) (1, 1); Ridge: gradient of w² is 2w (0, 0) (0.5, 1) (1, 2). Ridge's push is 2w, so it fades as the coefficient shrinks and a small coefficient is never quite driven to zero. Lasso's push is the same at 0.001 as at 50, a constant, which is what carries small coefficients all the way to zero and holds them there.\n\n## Which to use\n\n- **Ridge** when features are correlated and you want stability, or when you believe many features each contribute a little. Ridge handles correlated groups gracefully: it shares the coefficient across them rather than picking one.\n- **Lasso** when you believe most features are irrelevant and want a short list. Its weakness is exactly ridge's strength: given five correlated features, lasso keeps one essentially at random and zeroes the other four, which makes the selected list unstable across samples.\n- **Elastic net**: both penalties, mixed with a ratio parameter. It is the answer when you want lasso's sparsity but have correlated groups, because it tends to keep or drop the whole group together.\n\n```python\nfrom sklearn.linear_model import RidgeCV, LassoCV, ElasticNetCV\nimport numpy as np\nridge = RidgeCV(alphas=np.logspace(-3, 3, 13)).fit(X_tr, y_tr)\nridge.alpha_        # the chosen penalty\n```\n\nThe `CV` variants choose `alpha` by cross-validation, which is what you want. Searching alpha over a log scale, not a linear one, is essential: the useful range spans several orders of magnitude and a linear grid wastes almost all its points.\n\n## Two things that will bite you\n\n**Scale the features first, always.** The penalty is applied to the coefficient values, and a feature measured in rupees has a small coefficient purely because of its units. Without scaling, ridge penalises features by their unit of measurement rather than their contribution: the income coefficient is naturally tiny so it escapes the penalty, while the coefficient on a 0/1 flag is naturally larger and gets crushed. Put a `StandardScaler` before any penalised model and treat it as non-optional.\n\n**Do not penalise the intercept.** Every serious library already excludes it, but if you write the loss yourself, remember: penalising the intercept means the model is punished for the target having a nonzero mean, which is not a complexity you want to discourage.\n\n## The wider point\n\nRegularisation is not a linear-model technique. It is a general principle (constrain the model's freedom so it cannot fit noise) and every family has its version. Trees limit depth and require minimum samples per leaf. Neural networks use weight decay (which is exactly L2), dropout, and early stopping. Boosting uses a learning rate and subsampling.\n\nAll of them are the same trade in different clothes: accept worse training performance in exchange for a model that changes less when the data changes. Once you see that, `max_depth`, `alpha`, `dropout` and `n_estimators` stop being a list of unrelated knobs and become one knob with several handles."},"check":{"question":"A dataset has five nearly identical sensor readings, all genuinely predictive. Lasso keeps one and zeroes the other four; ridge keeps all five with smaller coefficients. Which behaviour is a problem, and why?","options":[{"text":"Ridge's, because keeping five collinear features inflates the standard errors and makes the model unstable.","correct":false,"tempting":"Correctly notes collinearity inflates coefficient uncertainty, but ridge was specifically designed to stabilise exactly this case, which is why it shares the weight."},{"text":"Lasso's, if you plan to read the selected features as 'the important ones', the choice among equally good correlated features is arbitrary and will differ on a resample.","correct":true},{"text":"Lasso's, because dropping four predictive sensors necessarily reduces predictive accuracy.","correct":false,"tempting":"Assumes the dropped features add predictive value, but with near-identical readings the retained one carries essentially the same information, so accuracy barely moves."},{"text":"Neither, since both minimise a convex objective and therefore reach the same unique solution.","correct":false,"tempting":"Both objectives are convex, but they are different objectives with different minima, so convexity does not imply they agree."}]},"prev":{"slug":"reading-coefficients","title":"What a coefficient says, and the four ways it lies","url":"https://addaly.com/learn/machine-learning-foundations/reading-coefficients"},"next":{"slug":"making-a-linear-model-curve","title":"Making a straight-line model bend","url":"https://addaly.com/learn/machine-learning-foundations/making-a-linear-model-curve"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Regularisation: paying for large coefficients', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/ridge-and-lasso (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Regularisation: paying for large coefficients","courseTitle":"Machine Learning, Foundations","module":{"index":2,"slug":"linear-models-and-fitting","title":"Linear models and the machinery of fitting"},"answer":null,"keyIdea":"Ridge and lasso add a price on coefficient size to the loss; lasso reaches exactly zero because the gradient of the absolute value stays constant as the coefficient shrinks, while ridge’s push fades away near zero.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"The push each penalty exerts on a coefficient","series":[{"label":"Lasso: gradient of |w| is constant","points":[[0.01,1],[1,1]]},{"label":"Ridge: gradient of w² is 2w","points":[[0,0],[0.5,1],[1,2]]}],"xLabel":"Coefficient size, w","yLabel":"Push towards zero","caption":"Ridge’s push is 2w, so it fades as the coefficient shrinks and a small coefficient is never quite driven to zero. Lasso’s push is the same at 0.001 as at 50, a constant, which is what carries small coefficients all the way to zero and holds them there."},"alt":"Figure: The push each penalty exerts on a coefficient. Line chart of Push towards zero against Coefficient size, w: Lasso: gradient of |w| is constant (0.01, 1) (1, 1); Ridge: gradient of w² is 2w (0, 0) (0.5, 1) (1, 2). Ridge’s push is 2w, so it fades as the coefficient shrinks and a small coefficient is never quite driven to zero. Lasso’s push is the same at 0.001 as at 50, a constant, which is what carries small coefficients all the way to zero and holds them there."}},"blocks":[{"t":"h2","id":"the-idea-in-one-sentence","runs":[{"text":"The idea in one sentence"}]},{"t":"p","runs":[{"text":"Add the size of the coefficients to the loss, so the optimiser has to justify every unit of coefficient it spends with a matching reduction in error."}]},{"t":"code","lang":"text","label":"Text","text":"ordinary:  minimise  sum of squared errors\nridge:     minimise  sum of squared errors + alpha * sum of w^2\nlasso:     minimise  sum of squared errors + alpha * sum of |w|"},{"t":"p","runs":[{"text":"That is the whole mechanism. "},{"text":"alpha","code":true},{"text":" is the price. At "},{"text":"alpha = 0","code":true},{"text":" you are back to ordinary least squares. As "},{"text":"alpha","code":true},{"text":" rises, coefficients shrink towards zero and the model gets simpler whether it likes it or not."}]},{"t":"h2","id":"why-shrinking-helps","runs":[{"text":"Why shrinking helps"}]},{"t":"p","runs":[{"text":"A model overfits by using large, opposing coefficients to thread through individual training points. Two correlated features with "},{"text":"+900","code":true},{"text":" and "},{"text":"-880","code":true},{"text":" can produce wild predictions from small input changes: a tiny shift in either feature moves the output enormously. That is a model with high variance: refit it on a different sample and it makes different predictions."}]},{"t":"p","runs":[{"text":"Making large coefficients expensive removes that option. The model must explain the data with modest weights or not at all. You give up a little training accuracy and buy a large reduction in how much the model changes between samples. That trade is worthwhile whenever you have many features relative to rows, and it is why ridge is a sensible default rather than a rescue."}]},{"t":"h2","id":"the-difference-between-l2-and-l1-mechanically","runs":[{"text":"The difference between L2 and L1, mechanically"}]},{"t":"p","runs":[{"text":"Everyone learns that lasso sets coefficients to exactly zero and ridge does not. Fewer learn why, and the reason is simple enough to hold."}]},{"t":"p","runs":[{"text":"Consider the penalty’s own gradient: the push it exerts on a coefficient."}]},{"t":"ul","items":[[{"text":"Ridge penalises "},{"text":"w^2","code":true},{"text":". Its gradient is "},{"text":"2w","code":true},{"text":". As "},{"text":"w","code":true},{"text":" approaches zero, the push approaches zero. The penalty loses interest in small coefficients, so they shrink and shrink and never quite arrive."}],[{"text":"Lasso penalises "},{"text":"|w|","code":true},{"text":". Its gradient is a constant (the sign of "},{"text":"w","code":true},{"text":") regardless of size. A coefficient of 0.001 gets exactly the same push toward zero as a coefficient of 50. That constant push is strong enough to drive small coefficients to exactly zero and hold them there."}]]},{"t":"p","runs":[{"text":"So lasso performs feature selection as a side effect of its penalty shape, not as a separate step. With "},{"text":"alpha","code":true},{"text":" tuned, a lasso on 200 features may keep 23 and zero the rest."}]},{"t":"h2","id":"which-to-use","runs":[{"text":"Which to use"}]},{"t":"ul","items":[[{"text":"Ridge","b":true},{"text":" when features are correlated and you want stability, or when you believe many features each contribute a little. Ridge handles correlated groups gracefully: it shares the coefficient across them rather than picking one."}],[{"text":"Lasso","b":true},{"text":" when you believe most features are irrelevant and want a short list. Its weakness is exactly ridge’s strength: given five correlated features, lasso keeps one essentially at random and zeroes the other four, which makes the selected list unstable across samples."}],[{"text":"Elastic net","b":true},{"text":": both penalties, mixed with a ratio parameter. It is the answer when you want lasso’s sparsity but have correlated groups, because it tends to keep or drop the whole group together."}]]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.linear_model import RidgeCV, LassoCV, ElasticNetCV\nimport numpy as np\nridge = RidgeCV(alphas=np.logspace(-3, 3, 13)).fit(X_tr, y_tr)\nridge.alpha_        # the chosen penalty"},{"t":"p","runs":[{"text":"The "},{"text":"CV","code":true},{"text":" variants choose "},{"text":"alpha","code":true},{"text":" by cross-validation, which is what you want. Searching alpha over a log scale, not a linear one, is essential: the useful range spans several orders of magnitude and a linear grid wastes almost all its points."}]},{"t":"h2","id":"two-things-that-will-bite-you","runs":[{"text":"Two things that will bite you"}]},{"t":"p","runs":[{"text":"Scale the features first, always.","b":true},{"text":" The penalty is applied to the coefficient values, and a feature measured in rupees has a small coefficient purely because of its units. Without scaling, ridge penalises features by their unit of measurement rather than their contribution: the income coefficient is naturally tiny so it escapes the penalty, while the coefficient on a 0/1 flag is naturally larger and gets crushed. Put a "},{"text":"StandardScaler","code":true},{"text":" before any penalised model and treat it as non-optional."}]},{"t":"p","runs":[{"text":"Do not penalise the intercept.","b":true},{"text":" Every serious library already excludes it, but if you write the loss yourself, remember: penalising the intercept means the model is punished for the target having a nonzero mean, which is not a complexity you want to discourage."}]},{"t":"h2","id":"the-wider-point","runs":[{"text":"The wider point"}]},{"t":"p","runs":[{"text":"Regularisation is not a linear-model technique. It is a general principle (constrain the model’s freedom so it cannot fit noise) and every family has its version. Trees limit depth and require minimum samples per leaf. Neural networks use weight decay (which is exactly L2), dropout, and early stopping. Boosting uses a learning rate and subsampling."}]},{"t":"p","runs":[{"text":"All of them are the same trade in different clothes: accept worse training performance in exchange for a model that changes less when the data changes. Once you see that, "},{"text":"max_depth","code":true},{"text":", "},{"text":"alpha","code":true},{"text":", "},{"text":"dropout","code":true},{"text":" and "},{"text":"n_estimators","code":true},{"text":" stop being a list of unrelated knobs and become one knob with several handles."}]}],"check":{"question":[{"text":"A dataset has five nearly identical sensor readings, all genuinely predictive. Lasso keeps one and zeroes the other four; ridge keeps all five with smaller coefficients. Which behaviour is a problem, and why?"}],"options":[{"text":[{"text":"Ridge’s, because keeping five collinear features inflates the standard errors and makes the model unstable."}],"correct":false,"tempting":[{"text":"Correctly notes collinearity inflates coefficient uncertainty, but ridge was specifically designed to stabilise exactly this case, which is why it shares the weight."}]},{"text":[{"text":"Lasso’s, if you plan to read the selected features as ‘the important ones’, the choice among equally good correlated features is arbitrary and will differ on a resample."}],"correct":true,"tempting":null},{"text":[{"text":"Lasso’s, because dropping four predictive sensors necessarily reduces predictive accuracy."}],"correct":false,"tempting":[{"text":"Assumes the dropped features add predictive value, but with near-identical readings the retained one carries essentially the same information, so accuracy barely moves."}]},{"text":[{"text":"Neither, since both minimise a convex objective and therefore reach the same unique solution."}],"correct":false,"tempting":[{"text":"Both objectives are convex, but they are different objectives with different minima, so convexity does not imply they agree."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"reading-coefficients","title":"What a coefficient says, and the four ways it lies","minutes":9},"next":{"slug":"making-a-linear-model-curve","title":"Making a straight-line model bend","minutes":8}}}