Logs, and why losses are logged

What a log is

A logarithm answers: what power do I raise the base to, to get this number?

log10(1000) = 3     because 10^3 = 1000
log10(0.001) = -3   because 10^-3 = 0.001

Machine learning mostly uses base e ≈ 2.718, written ln or just log in code. The base changes the scale, not the behaviour.

Two properties are the whole reason logs are everywhere.

Logs turn multiplication into addition. log(a × b) = log(a) + log(b). Products become sums.

Logs stretch out the small numbers. Between 0.1 and 0.01 there is barely any room on a normal ruler. In log terms they are a full unit apart, the same distance as 1 to 0.1.

The problem logs solve first

The probability a model assigns to a whole sentence is the probability of the first token, times the second given the first, times the third, and on. Multiply 500 numbers, each under 1.

0.1 ** 500   # 0.0 in float32. Not small. Zero.

The true value is around 10^-500, and a 32-bit float bottoms out near 10^-38. Your number is gone, and every comparison after it is meaningless. This is underflow, and it is not a corner case — it happens on every sentence.

Take logs and the product becomes a sum of 500 numbers each around -2.3. Total: about -1150. A float handles that without blinking. This alone would justify logs. The rest is a bonus.

Cross-entropy, read as a shape

The standard loss for classification and language modelling is -log(p), where p is the probability the model gave to the correct answer. Negative, because probabilities are below 1 so their logs are negative, and a loss should be positive.

Look at what that function does:

p = 0.99   loss = 0.01
p = 0.90   loss = 0.11
p = 0.50   loss = 0.69
p = 0.10   loss = 2.30
p = 0.01   loss = 4.61
p = 0.0001 loss = 9.21
p = 0      loss = infinity

Read the top and the bottom. Going from 0.90 to 0.99 saves you 0.10 of loss — a decent model getting slightly better barely moves the needle. Going from 0.01 to 0.10 saves you 2.31, twenty times as much.

The loss cares enormously about the cases the model is confidently wrong on, and hardly at all about the cases it already has. That is exactly the right priority, and it comes free from the shape of the log. A squared-error loss would treat 0.90 → 0.99 and 0.01 → 0.10 as similar improvements, which is not what you want from a model.

The infinity at zero is real and it matters. Assigning 0 probability to something that then happens is an infinite penalty, which is why models never fully rule anything out, and why a stray log(0) shows up as nan in your training run.

Reading a loss number

Cross-entropy in nats is hard to feel. Exponentiate it and you get perplexity:

loss 2.0  ->  e^2.0 = 7.4
loss 3.0  ->  e^3.0 = 20.1

A loss of 2.0 means the model is about as uncertain as someone guessing uniformly among 7.4 options at each token. That is a number you can hold.

It also explains a thing that puzzles people watching loss curves. Dropping from 4.0 to 3.0 looks identical on the plot to dropping from 3.0 to 2.0. In perplexity, the first is 54.6 down to 20.1 and the second is 20.1 down to 7.4. Each unit of loss is the same multiplicative gain, which is why a curve that looks like it has flattened out near the end is often still delivering real improvements. Log scales compress the top and stretch the bottom. That is their job, and it is why a flat-looking tail is not the same thing as a finished model.