Batches, epochs, and the two knobs that decide everything

Three ways to take one step

Gradient descent needs a gradient, and computing it means looking at data. How much data you look at before each step is a real choice with real consequences.

Full batch. Compute the gradient over every row, then take one step. The direction is the true average direction. On a million rows, one step costs a full pass over a million rows, so you might take 200 steps in an hour.

Stochastic (SGD). Compute the gradient from one row and step immediately. On a million rows you take a million steps per pass. Each direction is noisy (one row is a poor estimate of the average) but you take so many steps that the noise partly cancels and you get far more progress per unit of computation.

Mini-batch. Compute the gradient from 32, or 256, or 1,024 rows. This is what everything actually uses, because it gets most of SGD’s step count while the averaging across a few dozen rows cuts most of the noise, and because a batch of 256 rows is a matrix multiplication that hardware is built to do fast, whereas 256 separate single-row updates is 256 slow operations.

An epoch is one full pass over the training data. With 100,000 rows and a batch size of 250, an epoch is 400 steps.

The noise is doing something useful

The standard framing treats mini-batch noise as a cost you pay for speed. That is incomplete. The noise also helps you escape places you would otherwise be stuck in: a flat region or a poor local minimum can trap full-batch descent, while the jitter of a noisy gradient shakes the parameters out of it.

This is why very large batches sometimes generalise slightly worse than moderate ones, even at equal accuracy on the training set. It is an active area, the effect is modest, and the practical advice is unexciting: 32 to 512 covers almost everything, start at 128 if you have no reason to prefer another number.

The learning rate is the knob that matters

If you can tune one thing, tune this. The failure modes are distinctive and you can name them from a plot.

• Far too high. Loss becomes NaN within a few steps. The parameters overshot so far that the next gradient was larger, then larger again. This is divergence and it is unmistakable.
• Too high. Loss falls, then bounces around a floor it never gets below, sometimes rising. The optimiser is stepping across the valley rather than down it.
• About right. Loss falls quickly, then more slowly, then flattens.
• Too low. Loss falls in a nearly straight line and is still falling when you run out of patience. Slow, but not wrong, and this is the safe direction to err in.

A practical starting point for a scaled linear or logistic model is 0.01. For neural networks with Adam, 0.001 is the near-universal default. Then move in factors of three, 0.003, 0.01, 0.03, rather than in small increments; the useful range spans orders of magnitude.

Scaling and the learning rate are the same problem

This is the connection most courses leave implicit. Suppose one feature is age (18 to 80) and another is annual income (200,000 to 5,000,000). From the by-hand lesson, the push on a weight is proportional to the feature value. So the income weight receives gradients tens of thousands of times larger than the age weight.

One learning rate has to serve both. Set it small enough not to blow up the income weight and the age weight barely moves; set it large enough to move age and income diverges. The loss surface is a long narrow ravine and the optimiser zigzags down it.

Standardising both features to mean 0 and standard deviation 1 turns the ravine into something closer to a bowl, and the same optimiser converges in a fraction of the steps. This is why scaling is required for gradient-based models and irrelevant for trees: trees split on one feature at a time and never combine them in a weighted sum, so the units cancel out of the whole procedure.

Schedules: start big, end small

A constant learning rate has to be small enough for the endgame, which wastes the beginning. So decay it.

from sklearn.linear_model import SGDRegressor
SGDRegressor(learning_rate="invscaling", eta0=0.01, power_t=0.25)

The standard shapes are step decay (multiply by 0.1 every N epochs), cosine decay (a smooth fall to near zero), and warmup-then-decay (rise for the first few hundred steps, then fall), which is standard for transformers because early gradients on freshly initialised weights are unreliable and large early steps do damage.

A decaying rate also explains a shape you will see in loss curves: a visible cliff where the loss drops suddenly. That is usually not the model learning something; it is the learning rate stepping down and letting the optimiser settle into a place it had been bouncing around for an hour.

What to do first

If a model will not converge, the order to check is: scale the features, then lower the learning rate, then increase the iterations. Most people do that order backwards and spend an afternoon on it.