Regularisation is a prior: L2, L1 and the diamond

Where maximum likelihood goes wrong, and the repair

Module 5 derived every standard loss from maximum likelihood: choose the parameters that make the data most probable. It also noted the flaw. Given enough parameters and too little data, the parameters that make the training data most probable are ones that have memorised it. The likelihood is happy to set a weight to 40 if that fits one row.

The repair is to say what you believed about the parameters before seeing data. Bayes' rule, from module 4, gives

posterior ∝ likelihood × prior

and maximising the posterior instead of the likelihood is maximum a posteriori estimation, MAP. Take logs:

log posterior = log likelihood + log prior + const

Minimising the negative of this gives

loss = data loss + (−log prior)

The second term is the regulariser. Every penalty term you have seen added to a loss is a negative log prior, whether or not the person who added it thought of it that way.

A Gaussian prior is weight decay

Suppose you believe each weight is probably small: normally distributed around zero with spread τ. The log of that density is −w²/(2τ²) plus a constant, so the penalty is

Σ w² / (2τ²)

That is the L2 penalty, with strength λ = 1/(2τ²). A large λ is a narrow prior, a strong belief that weights are near zero. Weight decay in an optimiser is this penalty's gradient, λ w, applied each step. The hyperparameter you tune on a validation set is, in this reading, the width of your prior on the weights, chosen empirically. That is a legitimate way to choose a prior, and it is what nearly everyone does.

A Laplace prior is L1

Believe instead that weights follow a Laplace distribution, whose density is exp(−|w|/b). The log is −|w|/b, and the penalty is

Σ |w| / b

The L1 penalty. Same recipe, different prior, and a qualitatively different result: L1 sets some weights to exactly zero, L2 never does. The course machine-learning-foundations uses both; here is the mechanism.

Why L1 reaches zero and L2 does not

Take one weight whose data loss is (w − 0.3)²/2, pulling it toward 0.3 with unit curvature.

With an L2 penalty λw²/2 at λ = 1, the total is (w − 0.3)²/2 + w²/2. Its derivative, (w − 0.3) + w, is zero at w = 0.15. The weight is halved. Make λ larger and it shrinks further, but the derivative of the penalty is λw, which vanishes as w approaches zero. The penalty's pull fades exactly when it would need to finish the job. w is never exactly zero.

With an L1 penalty λ|w| at λ = 0.5, the pull is 0.5 toward zero regardless of how small w is. The data pull at w is 0.3 − w. If the data pull at w = 0 is weaker than 0.5, and it is, since 0.3 < 0.5, the weight cannot leave zero. So w = 0 exactly. At λ = 0.1 the data win and w = 0.3 − 0.1 = 0.2. The general solution is

w = sign(w₀) × max(0, |w₀| − λ)

called soft thresholding: shrink by λ, and if that crosses zero, stop there. Weights whose data pull is weaker than λ are removed. That is a constant force against a fading one, and it is the whole reason L1 produces sparse models.

The usual picture says the same thing geometrically. The L2 penalty's level sets are circles, the L1 penalty's are diamonds, and the minimum of loss plus penalty tends to touch a diamond at a corner, where a coordinate is zero. The picture and the derivative are one fact.

Early stopping is a prior too

Start gradient descent from zero weights and stop after t steps. Directions in which the loss has high curvature converge quickly; directions with low curvature have barely moved from zero. Stopping early therefore holds the low-curvature directions near zero, which is what an L2 penalty of strength about 1/(learning rate × t) would have done. The number of epochs is a regularisation strength, which is why it gets tuned on the validation set like one.

Two honest limits

MAP is the peak of the posterior, not its centre. Under a Laplace prior the peak is sparse, with weights at exactly zero; the posterior mean is not sparse at all. Sparsity is a property of the point estimate you chose to report, not of the belief. Averaging models trained with L1 loses the zeros.

The prior is a modelling choice with consequences. A Gaussian prior on weights says a weight of 3τ is rare and 6τ nearly impossible. If the true function needs one large weight, L2 will fight it in every step and settle for many small ones instead. Regularisation encodes what you expect the solution to look like, and when the expectation is wrong the fit is worse, not merely slower.

One line to try

import numpy as np
w0, lam = 0.3, 0.5
l2 = w0 / (1 + lam)                                  # 0.2 for lam=0.5
l1 = np.sign(w0) * max(0.0, abs(w0) - lam)            # 0.0

Change lam and watch L1 snap to zero while L2 only ever halves the distance.