Regularisation: paying for large coefficients

The idea in one sentence

Add the size of the coefficients to the loss, so the optimiser has to justify every unit of coefficient it spends with a matching reduction in error.

ordinary:  minimise  sum of squared errors
ridge:     minimise  sum of squared errors + alpha * sum of w^2
lasso:     minimise  sum of squared errors + alpha * sum of |w|

That is the whole mechanism. alpha is the price. At alpha = 0 you are back to ordinary least squares. As alpha rises, coefficients shrink towards zero and the model gets simpler whether it likes it or not.

Why shrinking helps

A model overfits by using large, opposing coefficients to thread through individual training points. Two correlated features with +900 and -880 can produce wild predictions from small input changes: a tiny shift in either feature moves the output enormously. That is a model with high variance: refit it on a different sample and it makes different predictions.

Making large coefficients expensive removes that option. The model must explain the data with modest weights or not at all. You give up a little training accuracy and buy a large reduction in how much the model changes between samples. That trade is worthwhile whenever you have many features relative to rows, and it is why ridge is a sensible default rather than a rescue.

The difference between L2 and L1, mechanically

Everyone learns that lasso sets coefficients to exactly zero and ridge does not. Fewer learn why, and the reason is simple enough to hold.

Consider the penalty’s own gradient: the push it exerts on a coefficient.

• Ridge penalises w^2. Its gradient is 2w. As w approaches zero, the push approaches zero. The penalty loses interest in small coefficients, so they shrink and shrink and never quite arrive.
• Lasso penalises |w|. Its gradient is a constant (the sign of w) regardless of size. A coefficient of 0.001 gets exactly the same push toward zero as a coefficient of 50. That constant push is strong enough to drive small coefficients to exactly zero and hold them there.

So lasso performs feature selection as a side effect of its penalty shape, not as a separate step. With alpha tuned, a lasso on 200 features may keep 23 and zero the rest.

Which to use

• Ridge when features are correlated and you want stability, or when you believe many features each contribute a little. Ridge handles correlated groups gracefully: it shares the coefficient across them rather than picking one.
• Lasso when you believe most features are irrelevant and want a short list. Its weakness is exactly ridge’s strength: given five correlated features, lasso keeps one essentially at random and zeroes the other four, which makes the selected list unstable across samples.
• Elastic net: both penalties, mixed with a ratio parameter. It is the answer when you want lasso’s sparsity but have correlated groups, because it tends to keep or drop the whole group together.

from sklearn.linear_model import RidgeCV, LassoCV, ElasticNetCV
import numpy as np
ridge = RidgeCV(alphas=np.logspace(-3, 3, 13)).fit(X_tr, y_tr)
ridge.alpha_        # the chosen penalty

The CV variants choose alpha by cross-validation, which is what you want. Searching alpha over a log scale, not a linear one, is essential: the useful range spans several orders of magnitude and a linear grid wastes almost all its points.

Two things that will bite you

Scale the features first, always. The penalty is applied to the coefficient values, and a feature measured in rupees has a small coefficient purely because of its units. Without scaling, ridge penalises features by their unit of measurement rather than their contribution: the income coefficient is naturally tiny so it escapes the penalty, while the coefficient on a 0/1 flag is naturally larger and gets crushed. Put a StandardScaler before any penalised model and treat it as non-optional.

Do not penalise the intercept. Every serious library already excludes it, but if you write the loss yourself, remember: penalising the intercept means the model is punished for the target having a nonzero mean, which is not a complexity you want to discourage.

The wider point

Regularisation is not a linear-model technique. It is a general principle (constrain the model’s freedom so it cannot fit noise) and every family has its version. Trees limit depth and require minimum samples per leaf. Neural networks use weight decay (which is exactly L2), dropout, and early stopping. Boosting uses a learning rate and subsampling.

All of them are the same trade in different clothes: accept worse training performance in exchange for a model that changes less when the data changes. Once you see that, max_depth, alpha, dropout and n_estimators stop being a list of unrelated knobs and become one knob with several handles.