Loss functions are not arbitrary: maximum likelihood

Why squared error, and not cubed

Nobody explains where the loss functions come from. Squared error for regression, cross-entropy for classification: they are presented as conventions, and a learner reasonably wonders why not absolute error, or the fourth power, or something else. There is an answer, and it is the same answer for every loss you will meet.

The principle is maximum likelihood. State how you believe the data were generated, as a probability distribution with some unknown parameters. Then choose the parameters that make the data you actually observed as probable as possible. The loss is minus the log of that probability. Every standard loss is this recipe applied to a particular assumption about the noise.

Squared error is a Gaussian assumption

Suppose you believe each target is the model's prediction plus Gaussian noise of fixed spread σ:

y = f(x) + noise,   noise ~ Normal(0, σ²)

The probability density of observing y is then

p(y | x) = (1 / (σ√(2π))) × exp(−(y − f(x))² / (2σ²))

Take the log:

log p(y | x) = −(y − f(x))² / (2σ²) − log(σ√(2π))

The second term does not depend on the model. Maximising the log-likelihood over many independent examples means maximising the sum of the first terms, which means minimising the sum of squared errors. Mean squared error is not a convention. It is the statement "I believe the errors are Gaussian with constant variance", and the squaring is the shape of the Gaussian's exponent.

Do it with numbers once. Three points with targets 2.0, 4.1 and 5.9, and two candidate models. Model A predicts 2, 4, 6; model B predicts 2.5, 4.5, 5.5. With σ = 1:

A: squared errors 0.00 + 0.01 + 0.01 = 0.02  →  log-lik = −0.01 + const
B: squared errors 0.25 + 0.16 + 0.16 = 0.57  →  log-lik = −0.285 + const

A is e^(0.275) = 1.32 times as likely to have produced the data. Smaller squared error and larger likelihood are the same fact.

Change the assumption, change the loss

• Laplace noise, whose density is exp(−|y − f| / b), gives absolute error. The Laplace has fatter tails than the Gaussian, so a large error is merely unlikely rather than astronomically so, and the fit is not dragged toward it. This is why L1 loss is robust to outliers: it is the loss that expected them.
• Bernoulli outcomes, a label that is 1 with probability q and 0 otherwise, give −(y log q + (1 − y) log(1 − q)), which is binary cross-entropy.
• Categorical outcomes give −log q(correct class), the cross-entropy of the previous lesson.
• Poisson counts, for something like the number of clicks in an hour, give f − y log f, the Poisson loss, and the model predicts a log-rate.

The method also explains a mistake you will see: training a house-price model with squared error when a few mansions are in the data. The Gaussian assumption says an error ten times the typical one has probability e^(−50), so the fit distorts itself to avoid it. If your errors are not Gaussian, the loss has been told a falsehood, and it will act on it.

One gradient for all of them

A striking consequence. For every loss in the list above, the derivative of the loss with respect to the model's raw output is the same expression:

prediction − target

For squared error it is f(x) − y directly. For binary cross-entropy through a sigmoid it is q − y. For softmax cross-entropy it is p_i − y_i, which the lesson on softmax computes explicitly. Linear regression, logistic regression and a softmax classifier are the same algorithm with three different noise assumptions, and their training loops differ by one line.

What maximum likelihood gets wrong

It believes the data too much. Observe an event three times out of three and maximum likelihood says its probability is exactly 1: the value that makes three-for-three most probable is certainty. A model with many parameters and few examples will, in the same way, choose parameters that make the training set look inevitable, which is the mechanism behind overfitting. The fix is to add a prior, a statement of what parameters were plausible before seeing data, and that turns out to be what regularisation is. The next module derives it.

It also assumes the examples are independent. Two hundred rows from the same customer are not two hundred pieces of evidence, but the product of their likelihoods treats them as such.

The variance you dropped

In the Gaussian derivation, σ was fixed and fell out of the loss. If instead you let the model predict its own σ(x) for each input, the log σ term stays, and the loss becomes

(y − f(x))² / (2σ(x)²) + log σ(x)

The model can now admit that some inputs are harder to predict, by widening σ there, and it is charged log σ for doing so, which stops it widening everywhere. That is heteroscedastic regression, and it is one line of extra algebra away from the ordinary kind. Once you see losses as log-likelihoods, extensions like this stop looking like tricks.