What a log is, and why the axis got logged

A log counts multiplications

A logarithm answers one question: how many times must I multiply the base by itself to reach this number? log2(1024) = 10 because ten doublings reach 1,024. log10(1,000,000) = 6 because six tens do. That is the whole definition. Everything else is consequence.

Three bases matter in this field.

• Base 2 counts doublings and gives answers in bits. log2(32000) = 14.97, so choosing one token from a 32,000-word vocabulary is about fifteen yes-or-no questions.
• Base 10 counts orders of magnitude. A learning rate of 0.001 sits at log10 = -3.
• Base e, written ln, where e = 2.71828. It is the one calculus prefers, because the slope of e^x is e^x. Every loss you will see from PyTorch is in this base, in units called nats.

You convert between them by one multiplication: ln x = 2.303 × log10 x and log2 x = 1.443 × ln x. Remember ln 2 = 0.693 and ln 10 = 2.303; they come up constantly.

The four rules, and the one that matters

log(a × b) = log a + log b
log(a / b) = log a − log b
log(a^k)   = k × log a
log_b(x)   = ln x / ln b

The first rule is the reason logs exist in machine learning. A model that assigns probability 0.9 to each of 200 tokens in a sentence assigns the sentence 0.9^200 = 7 × 10^-10. Multiply another hundred such sentences together and the product is below anything a computer can store. Take logs first and the product becomes a sum: 200 × ln 0.9 = −21.07. Sums of moderate numbers stay moderate. Module 8 returns to exactly how small a number can get before it becomes zero.

Why the axis got logged

Take a learning-rate sweep over 0.00001, 0.0001, 0.001, 0.01 and 0.1. On a linear axis the first four values share the leftmost pixel and the fifth sits alone at the right. On a log axis they are evenly spaced, because equal spacing on a log axis means equal ratio, not equal difference. Each grid line is ten times the last.

The same applies to loss curves. A loss that falls from 8 to 4 in the first hundred steps and from 2.1 to 2.0 in the last ten thousand is invisible in its late phase on a linear plot. On a log y-axis the late phase is still readable, and something else appears: a curve that is a straight line on log-log axes is a power law, and one that is straight on a log-linear axis is an exponential. The last lesson of this module teaches you to read the slope. For now, the habit: if the quantity spans more than two orders of magnitude, log the axis.

import matplotlib.pyplot as plt
plt.plot(steps, loss)
plt.yscale("log")   # one line; nothing else changes

Two things a log axis cannot show

Zero. log(0) is minus infinity. A curve that reaches zero loss leaves the bottom of a log plot rather than touching an axis, and a bar chart with a zero bar has no bar. If your data contain zeros, plotting log(x + 1) is the usual compromise; say so on the axis label, because it distorts the small values.

Negatives. The log of a negative number is not a real number. Losses are non-negative, so this rarely bites there, but a log axis on a plot of gradients or weight changes silently drops every negative point.

A related trap is the epsilon people add to avoid the zero: log(p + 1e-9). It works, but log(1e-9) = −20.7, so a probability the model put at exactly zero is scored as a surprise of 20.7 nats rather than infinity. That is a choice, and it changes the average. Pick the epsilon consciously and keep it the same across every run you intend to compare.

Everyday logs you already read

Decibels are 10 × log10 of a power ratio, so 30 dB is a thousandfold. Earthquake magnitude is base 10, so a magnitude 7 releases about 32 times the energy of a 6. pH is a negative log. The screen brightness slider on most phones is logarithmic because perception is. You have been reading log scales for years; the only new thing is doing so on purpose.

Tools

Python gives you math.log (base e), math.log2, math.log10 and math.log(x, base). NumPy applies them to whole arrays. A phone calculator has log and ln. Nothing here needs a GPU or a paid licence.

One exercise, done once, fixes the idea: compute ln(32000) on a calculator, divide by ln 2, and confirm you get 14.97. Then compute a model's reported loss of 2.0 nats in bits: 2.0 × 1.443 = 2.89. That model needs just under three bits per token, against fifteen for a uniform guess. The next lessons say what those bits mean.