Addaly is in open beta. Things will change, and AI answers can be wrong — check anything that matters.

The Maths You Actually Need

Eight ideas that carry almost all the weight in machine learning.

Lesson 68 of 769 min

bf16, fp16, and why training keeps a float32 copy

Two ways to spend sixteen bits

Halve the width of a float and you must give up either range or precision. The two sixteen-bit formats make opposite choices.

format   sign  exponent  mantissa   max value   epsilon    digits
fp16      1       5        10        65,504     9.8e-4     ~3
bf16      1       8         7       3.4e38      7.8e-3     ~2
float32   1       8        23       3.4e38      1.2e-7     ~7

fp16 keeps ten bits of mantissa, three decimal digits, and spends only five on the exponent. Its largest value is 65,504 and its smallest normal value is 6 × 10^-5. bf16, "brain float", keeps float32's eight exponent bits and therefore its whole range, and pays with a seven-bit mantissa: two and a bit decimal digits. A bf16 number is a float32 with its bottom sixteen bits cut off, which makes conversion a truncation rather than a computation.

Module 7 established why either is wanted: half the bytes per parameter, twice the throughput on hardware built for them. The question is which half of the information to keep.

Why range beats precision for training

Gradients span an enormous range. Early layers of a deep network routinely see gradients of 10^-8, and activations after a bad step can reach 10^5. In fp16 the first underflows to zero and the second overflows to infinity, and neither problem announces itself except as a model that does not learn or a loss that becomes NaN. Module 5's overflow point moves from exp(88.7) to exp(11.1): a logit of 12 is enough to produce infinity in fp16.

bf16 has none of these problems, because its range is float32's. What it lacks is precision, and it turns out that a weight update does not need much: a gradient known to two digits still points the right way. So bf16 became the default for training wherever the hardware supports it, and fp16 survives mainly for inference on older cards.

Sixteen bits, spent two waysfp16: 5 exponent bits, 10 mantissa bitsLargest value 65,504Smallest normal value 0.00006About three decimal digitsA logit of 12 already overflows expTraining survives only with dynamic lossscalingbf16: 8 exponent bits, 7 mantissa bitsLargest value 3.4 followed by 38 zerosExactly float32's rangeAbout two decimal digitsConverting from float32 is a truncationThe default for training wherever hardwareallowsA weight update does not need many digits; it needs to be representable at all, which is why rangebeat precision. It is also why the optimiser keeps a float32 master copy: near 1 the gap between bf16numbers is 0.0078, so an update of 0.0001 added there vanishes entirely.
Sixteen bits, spent two waysfp16: 5 exponent bits, 10 mantissa bitsLargest value 65,504Smallest normal value 0.00006About three decimal digitsA logit of 12 already overflows expTraining survives only with dynamic lossscalingbf16: 8 exponent bits, 7 mantissa bitsLargest value 3.4 followed by 38 zerosExactly float32's rangeAbout two decimal digitsConverting from float32 is a truncationThe default for training wherever hardwareallowsA weight update does not need many digits; it needsto be representable at all, which is why range beatprecision. It is also why the optimiser keeps afloat32 master copy: near 1 the gap between bf16numbers is 0.0078, so an update of 0.0001 addedthere vanishes entirely.

Loss scaling: the fp16 workaround

Where fp16 must be used for training, the underflow is handled by multiplying the loss by a large constant, say 1,024, before the backward pass. Every gradient is then 1,024 times larger, which lifts the 10^-8 gradients into fp16's range. The optimiser divides by the same constant before applying the update. If any gradient overflows to infinity, the step is skipped and the scale is halved; if a run of steps succeeds, the scale is doubled. This is dynamic loss scaling, it is in every mixed-precision training loop that uses fp16, and its entire purpose is to fit a range of 10^13 into a format that holds 10^9.

Why there is a float32 copy

Here is the fact that explains the 16 bytes per parameter in module 7. Suppose a weight is 1.0 and the update is 0.0001. In bf16, with its two digits, 1.0 + 0.0001 = 1.0: the update is smaller than the gap between neighbouring bf16 numbers near 1, which is 0.0078. The weight never changes. In fp16 the gap near 1 is 0.001, and the same update is lost.

Small updates are the normal case late in training, when the learning rate has decayed. So the optimiser keeps a master copy of every weight in float32, applies the update there, where the gap near 1 is 10^-7, and then rounds the result to bf16 for the next forward pass. The half-precision weights are a working copy; the float32 ones are the truth. Remove the master copy and training stalls once the updates fall below the half-precision gap, which is typically within the first few thousand steps.

Where the precision is actually spent

Mixed precision is precise about what runs in what:

  • Matrix multiplies: inputs in bf16, but the products accumulated in float32 inside the hardware, then rounded once at the end. Accumulating a 4,096-term dot product in bf16 would lose most of it, for the reasons the next lesson gives.
  • Softmax, layer normalisation, the loss: computed in float32, because they involve sums of many terms and subtractions of nearly equal ones.
  • Weights and activations in memory: bf16.
  • Master weights, optimiser moments: float32.

The framework's autocast does this routing for you. What it cannot do is make a computation that you wrote in bf16 by hand, an accumulating sum or a variance, come out right.

Inference is more forgiving

No updates, no gradients, no accumulation across steps. Weights and activations in fp16 or bf16 lose almost nothing measurable, which is why every served model is at most half precision and usually smaller, and why the quantisation lesson can go further still. The one inference computation that still wants float32 is the final softmax over a large vocabulary, where bf16's two digits turn a distribution over 100,000 tokens into something visibly lumpy.

Seeing the gap

python
import torch
w = torch.tensor(1.0, dtype=torch.bfloat16)
print(w + 0.001)              # 1.0     the update vanished
w32 = torch.tensor(1.0)
print((w32 + 0.001).to(torch.bfloat16))   # 1.0 again after rounding, but 1.001 was kept in w32

The second line is the master copy in miniature: the float32 value remembers what the bf16 one cannot, and after ten such updates it has moved to 1.01, which bf16 can then represent.

The one thing to keep

fp16 keeps three digits and a range to 65,504 while bf16 keeps two digits and float32's full range, and because a small update added to a half-precision weight rounds away entirely, training keeps a float32 master copy of every weight and accumulates matrix products in float32.

Before you move on

A team trains in pure bf16 with no float32 master weights. Training looks normal for a few thousand steps, then the loss flattens and the weights stop changing though the gradients are non-zero. Why?

Pick the one you would defend. Nobody sees your answer.

No ads. No data sale. No public scores on people. Ever.

© 2026 Addaly