Exponentials, decay, and the memory of a moving average
Repeated multiplication has a shape
Multiply 0.9 by itself and watch what happens.
0.9^1 = 0.900
0.9^5 = 0.590
0.9^10 = 0.349
0.9^22 = 0.098
0.9^44 = 0.010Every ten steps the value falls by roughly the same factor, about 0.35. That is exponential decay, and it has a half-life: the number of steps to halve. For a factor β the half-life is ln 0.5 / ln β. For 0.9 that is −0.693 / −0.105 = 6.6 steps. For 0.99 it is 69 steps; for 0.999 it is 693. The half-life of β close to 1 is approximately 0.693 / (1 − β), a rule worth memorising.
Growth is the mirror image. 1.01^100 = 2.70, which is e to two decimal places, and this is not a coincidence: (1 + 1/n)^n approaches e as n grows, which is why e turns up whenever something compounds continuously.
The exponential moving average
Optimisers, loss smoothers and reinforcement learners all keep a running average that forgets the past gradually rather than all at once:
m_t = β × m_(t−1) + (1 − β) × g_tUnroll it once and the structure shows. The newest value g_t has weight (1 − β). The one before has weight (1 − β) × β, the one before that (1 − β) × β^2, and a value k steps back has weight (1 − β) × β^k. The weights are a decaying exponential, and they sum to 1.
How far back does the average remember? The weights form a geometric series whose mean position is β / (1 − β), and the useful rule is that the effective window is about 1 / (1 − β). For β = 0.9 that is ten steps. For β = 0.999 it is a thousand.
This is where Adam's two constants come from. Its first moment uses β1 = 0.9, a ten-step average of the gradient. Its second moment uses β2 = 0.999, a thousand-step average of the squared gradient. The second needs a long memory because it is estimating a variance, and variance estimates from ten samples are noisy in a way that would make the step size lurch.
The bias at the start, and the correction
Start the average at m_0 = 0. After one step, m_1 = (1 − β) × g_1. With β2 = 0.999 that is 0.001 × g_1: the average claims the squared gradient is a thousand times smaller than it is. Adam divides by the square root of this number to set the step, so an uncorrected first step would be about 30 times too large.
The correction divides by the total weight that has actually been applied, 1 − β^t. At t = 1 that is 1 − 0.999 = 0.001, so the correction multiplies by a thousand and cancels the bias exactly. At t = 1000 it is 1 − 0.999^1000 = 1 − 0.368 = 0.632, still a noticeable 1.6 times. By t = 5000 it is 0.993 and has faded. If you have ever wondered why Adam's code has two lines called bias_correction, that is the entire reason.
Schedules are exponentials too
An exponential learning-rate schedule multiplies by a factor γ every step. Over 50,000 steps at γ = 0.9999:
0.9999^50000 = e^(50000 × ln 0.9999) = e^(−5.0) = 0.0067So the final learning rate is 0.67 per cent of the initial one. If that seems too aggressive, the arithmetic tells you the fix: γ = 0.99995 gives e^(−2.5) = 0.082. Working in the exponent is easier than guessing at the factor.
The same shape is the discount in reinforcement learning. A discount of 0.99 per step gives rewards a horizon of about a hundred steps; beyond that they contribute less than a third of their face value, and beyond 460 steps less than one per cent.
Where the average misleads
An exponential average lags. With β = 0.99 a sudden spike in the loss appears in the smoothed curve about a hundred steps late and a hundred times smaller, and a genuine step change is drawn as a gentle slope. The smoothing slider in TensorBoard is exactly this average, and at its default setting it hides the variance that would tell you the learning rate is too high. Look at the raw curve at least once per run.
The start is biased as well. Unless the tool applies the correction above, the first hundred points of a smoothed curve are pulled toward zero, which looks like an implausibly good early loss. It is an artefact, not a result.
Tools
Everything here is arithmetic you can do on a calculator with ln and e^x, or in three lines of Python:
import math
beta = 0.999
print(1 / (1 - beta), math.log(0.5) / math.log(beta)) # window, half-lifeThe one thing to keep
An exponential moving average with factor β remembers roughly 1/(1−β) steps and weights the past geometrically, which is where Adam's 0.9 and 0.999 come from, why it needs a bias correction at the start, and why a smoothed loss curve shows a spike late and small.
Before you move on
Adam keeps its second-moment estimate with β2 = 0.999. Roughly how far back does that average remember, and how?
Pick the one you would defend. Nobody sees your answer.