Precision, and why bf16 ended a class of bugs

Three ways to store a number

A floating-point number is a sign bit, some exponent bits and some mantissa bits. The exponent sets the range: how large and how small a value can be. The mantissa sets the precision: how finely values are distinguished within that range.

• fp32: 1 sign, 8 exponent, 23 mantissa. Range to about 3 × 10³⁸, roughly seven decimal digits of precision.
• fp16: 1, 5, 10. Maximum value 65,504. Smallest normal value about 6 × 10⁻⁵. Three decimal digits.
• bf16: 1, 8, 7. The same exponent range as fp32, with only three mantissa bits’ worth of precision more than a rounding error.

That comparison is the whole story. bf16 gave up precision to keep range, and for neural network training range turned out to matter enormously and precision hardly at all.

What went wrong with fp16

Gradients in a transformer are small. Many of them are around 10⁻⁷ or smaller, which is below fp16’s smallest normal value, so they round to zero. A gradient that rounds to zero is a parameter that does not learn.

The workaround is loss scaling: multiply the loss by a large constant, say 2¹⁶, before the backward pass. Every gradient is scaled up by the same factor, lifting the small ones into representable range. Then divide the gradients by the same constant before the optimiser step.

This works, and it introduces a new failure. If the scale is too large, some gradients exceed 65,504 and become infinity. Frameworks handle this with dynamic loss scaling: start high, and when an overflow is detected, skip that optimiser step and halve the scale. So a healthy fp16 run periodically throws away steps. If you see log lines about the gradient scaler reducing its scale, that is normal. If you see it every few steps, your run is spending a meaningful fraction of its compute on nothing.

What bf16 fixed

With the exponent range of fp32, no gradient underflows and none overflows. No loss scaling, no skipped steps, no NaN at step 300 of a run you paid for. The lost mantissa bits do not matter because the optimiser accumulates in fp32 anyway.

This is why a great deal of pre-2023 training advice about NaN, loss spikes and scaler tuning is obsolete: it was fp16 advice.

The hardware line that decides for you

bf16 requires Ampere or newer. Concretely:

• Has bf16: A100, H100, L4, L40S, A10G, RTX 30-series and 40-series, and newer.
• Does not: the T4, which is Turing. And the T4 is exactly what free Google Colab and Kaggle give you.

So on a free tier you are on fp16, with loss scaling, and a divergence at 2e-4 that would have been fine on an A100 is a real possibility. The mitigations on a T4 are: keep the learning rate at the low end, do not skip warmup, and if you see repeated scaler reductions, drop the learning rate rather than investigating anything cleverer.

Check it in one line before you configure anything:

import torch
print(torch.cuda.get_device_name(0), torch.cuda.is_bf16_supported())

Mixed precision means mixed

“bf16 training” does not mean everything is bf16. The standard arrangement keeps an fp32 master copy of the weights and does the optimiser update in fp32, while forward and backward run in bf16. The reason is accumulation: adding a tiny update to a large weight in bf16 loses the update entirely, because the result rounds back to the original value.

This is also where the 16-bytes-per-parameter figure for full fine-tuning comes from: 4 bytes of fp32 master weights, 4 of fp32 gradients, 8 of AdamW state, with the bf16 working copies on top.

For QLoRA the arrangement differs again: base weights are 4-bit, the compute dtype for matrix multiplications is bf16, and only the small adapter is kept and updated in higher precision. That is three precisions in one model, which is normal and not a sign of anything wrong.

fp8 and lower

H100 and newer support fp8 for training, with per-tensor scaling to manage the tiny range. It is real and it is used at scale, but the tooling is still specialist and the gains show up on large pretraining runs rather than on a LoRA over 3,000 examples. For the work this course describes, bf16 where available and fp16 where not is the whole decision.

One caveat worth keeping: a model trained in bf16 and then served in fp16 can behave slightly differently, because a value near the edge of fp16’s range that was fine during training now saturates. It is rare, and when it happens it looks like a serving bug rather than a precision bug, which is why it takes so long to find.