{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"precision-and-mixed-precision","title":"Precision, and why bf16 ended a class of bugs","url":"https://addaly.com/learn/fine-tuning/precision-and-mixed-precision","position":15,"of":84,"minutes":9,"answer":"bf16 keeps fp32's exponent range and sacrifices mantissa bits, which removes the underflow that forced fp16 to use loss scaling, and since the free T4 has no bf16.","keyIdea":"bf16 keeps fp32's exponent range and sacrifices mantissa bits, which removes the underflow that forced fp16 to use loss scaling, and since the free T4 has no bf16, free-tier runs still live with the scaler and its skipped steps."},"body":{"format":"markdown","text":"## Three ways to store a number\n\nA floating-point number is a sign bit, some exponent bits and some mantissa bits. The exponent sets the *range*: how large and how small a value can be. The mantissa sets the *precision*: how finely values are distinguished within that range.\n\n- **fp32**: 1 sign, 8 exponent, 23 mantissa. Range to about 3 × 10³⁸, roughly seven decimal digits of precision.\n- **fp16**: 1, 5, 10. Maximum value **65,504**. Smallest normal value about 6 × 10⁻⁵. Three decimal digits.\n- **bf16**: 1, 8, 7. The *same exponent range as fp32*, with only three mantissa bits' worth of precision more than a rounding error.\n\nThat comparison is the whole story. bf16 gave up precision to keep range, and for neural network training range turned out to matter enormously and precision hardly at all.\n\n## What went wrong with fp16\n\nGradients in a transformer are small. Many of them are around 10⁻⁷ or smaller, which is below fp16's smallest normal value, so they round to zero. A gradient that rounds to zero is a parameter that does not learn.\n\nThe workaround is **loss scaling**: multiply the loss by a large constant, say 2¹⁶, before the backward pass. Every gradient is scaled up by the same factor, lifting the small ones into representable range. Then divide the gradients by the same constant before the optimiser step.\n\nThis works, and it introduces a new failure. If the scale is too large, some gradients exceed 65,504 and become infinity. Frameworks handle this with *dynamic* loss scaling: start high, and when an overflow is detected, skip that optimiser step and halve the scale. So a healthy fp16 run periodically throws away steps. If you see log lines about the gradient scaler reducing its scale, that is normal. If you see it every few steps, your run is spending a meaningful fraction of its compute on nothing.\n\n## What bf16 fixed\n\nWith the exponent range of fp32, no gradient underflows and none overflows. No loss scaling, no skipped steps, no NaN at step 300 of a run you paid for. The lost mantissa bits do not matter because the optimiser accumulates in fp32 anyway.\n\nThis is why a great deal of pre-2023 training advice about NaN, loss spikes and scaler tuning is obsolete: it was fp16 advice.\n\n## The hardware line that decides for you\n\nbf16 requires Ampere or newer. Concretely:\n\n- **Has bf16**: A100, H100, L4, L40S, A10G, RTX 30-series and 40-series, and newer.\n- **Does not**: the **T4**, which is Turing. And the T4 is exactly what free Google Colab and Kaggle give you.\n\nSo on a free tier you are on fp16, with loss scaling, and a divergence at 2e-4 that would have been fine on an A100 is a real possibility. The mitigations on a T4 are: keep the learning rate at the low end, do not skip warmup, and if you see repeated scaler reductions, drop the learning rate rather than investigating anything cleverer.\n\nCheck it in one line before you configure anything:\n\n```python\nimport torch\nprint(torch.cuda.get_device_name(0), torch.cuda.is_bf16_supported())\n```\n\n## Mixed precision means mixed\n\n\"bf16 training\" does not mean everything is bf16. The standard arrangement keeps an **fp32 master copy** of the weights and does the optimiser update in fp32, while forward and backward run in bf16. The reason is accumulation: adding a tiny update to a large weight in bf16 loses the update entirely, because the result rounds back to the original value.\n\nThis is also where the 16-bytes-per-parameter figure for full fine-tuning comes from: 4 bytes of fp32 master weights, 4 of fp32 gradients, 8 of AdamW state, with the bf16 working copies on top.\n\nFor QLoRA the arrangement differs again: base weights are 4-bit, the compute dtype for matrix multiplications is bf16, and only the small adapter is kept and updated in higher precision. That is three precisions in one model, which is normal and not a sign of anything wrong.\n\n## fp8 and lower\n\nH100 and newer support fp8 for training, with per-tensor scaling to manage the tiny range. It is real and it is used at scale, but the tooling is still specialist and the gains show up on large pretraining runs rather than on a LoRA over 3,000 examples. For the work this course describes, bf16 where available and fp16 where not is the whole decision.\n\nOne caveat worth keeping: a model *trained* in bf16 and then *served* in fp16 can behave slightly differently, because a value near the edge of fp16's range that was fine during training now saturates. It is rare, and when it happens it looks like a serving bug rather than a precision bug, which is why it takes so long to find."},"check":{"question":"A fine-tune on a free Colab T4 runs fine for 200 steps, then the logs fill with the gradient scaler halving its scale, and progress stalls. What is the mechanism?","options":[{"text":"The T4 has no bf16, so fp16 loss scaling is in use and overflows are skipping optimiser steps.","correct":true},{"text":"The T4's 16 GB of memory has filled, and the scaler is reducing precision to free space.","correct":false,"tempting":"Conflates the loss scaler with a memory-management mechanism; the scaler adjusts a multiplier on the loss and frees nothing."},{"text":"The cosine schedule has decayed the learning rate far enough that gradients now underflow in fp16.","correct":false,"tempting":"Inverts the cause: the scaler halves its scale in response to overflow to infinity, not underflow, and lower learning rates do not shrink gradients."},{"text":"bf16 was requested but silently fell back to fp32, so memory pressure is forcing the framework to rescale.","correct":false,"tempting":"Describes a fallback that would raise memory use without involving the gradient scaler at all, which only exists on the fp16 path."}]},"prev":{"slug":"batch-size-and-accumulation","title":"Batch size, accumulation and the token count","url":"https://addaly.com/learn/fine-tuning/batch-size-and-accumulation"},"next":{"slug":"where-the-memory-goes","title":"Where the memory actually goes","url":"https://addaly.com/learn/fine-tuning/where-the-memory-goes"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Precision, and why bf16 ended a class of bugs', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/precision-and-mixed-precision (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Precision, and why bf16 ended a class of bugs","courseTitle":"Fine-Tuning, and When Not To","module":{"index":2,"slug":"how-training-works","title":"How training actually works"},"answer":null,"keyIdea":"bf16 keeps fp32’s exponent range and sacrifices mantissa bits, which removes the underflow that forced fp16 to use loss scaling, and since the free T4 has no bf16, free-tier runs still live with the scaler and its skipped steps.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"three-ways-to-store-a-number","runs":[{"text":"Three ways to store a number"}]},{"t":"p","runs":[{"text":"A floating-point number is a sign bit, some exponent bits and some mantissa bits. The exponent sets the "},{"text":"range","i":true},{"text":": how large and how small a value can be. The mantissa sets the "},{"text":"precision","i":true},{"text":": how finely values are distinguished within that range."}]},{"t":"ul","items":[[{"text":"fp32","b":true},{"text":": 1 sign, 8 exponent, 23 mantissa. Range to about 3 × 10³⁸, roughly seven decimal digits of precision."}],[{"text":"fp16","b":true},{"text":": 1, 5, 10. Maximum value "},{"text":"65,504","b":true},{"text":". Smallest normal value about 6 × 10⁻⁵. Three decimal digits."}],[{"text":"bf16","b":true},{"text":": 1, 8, 7. The "},{"text":"same exponent range as fp32","i":true},{"text":", with only three mantissa bits’ worth of precision more than a rounding error."}]]},{"t":"p","runs":[{"text":"That comparison is the whole story. bf16 gave up precision to keep range, and for neural network training range turned out to matter enormously and precision hardly at all."}]},{"t":"h2","id":"what-went-wrong-with-fp16","runs":[{"text":"What went wrong with fp16"}]},{"t":"p","runs":[{"text":"Gradients in a transformer are small. Many of them are around 10⁻⁷ or smaller, which is below fp16’s smallest normal value, so they round to zero. A gradient that rounds to zero is a parameter that does not learn."}]},{"t":"p","runs":[{"text":"The workaround is "},{"text":"loss scaling","b":true},{"text":": multiply the loss by a large constant, say 2¹⁶, before the backward pass. Every gradient is scaled up by the same factor, lifting the small ones into representable range. Then divide the gradients by the same constant before the optimiser step."}]},{"t":"p","runs":[{"text":"This works, and it introduces a new failure. If the scale is too large, some gradients exceed 65,504 and become infinity. Frameworks handle this with "},{"text":"dynamic","i":true},{"text":" loss scaling: start high, and when an overflow is detected, skip that optimiser step and halve the scale. So a healthy fp16 run periodically throws away steps. If you see log lines about the gradient scaler reducing its scale, that is normal. If you see it every few steps, your run is spending a meaningful fraction of its compute on nothing."}]},{"t":"h2","id":"what-bf16-fixed","runs":[{"text":"What bf16 fixed"}]},{"t":"p","runs":[{"text":"With the exponent range of fp32, no gradient underflows and none overflows. No loss scaling, no skipped steps, no NaN at step 300 of a run you paid for. The lost mantissa bits do not matter because the optimiser accumulates in fp32 anyway."}]},{"t":"p","runs":[{"text":"This is why a great deal of pre-2023 training advice about NaN, loss spikes and scaler tuning is obsolete: it was fp16 advice."}]},{"t":"h2","id":"the-hardware-line-that-decides-for-you","runs":[{"text":"The hardware line that decides for you"}]},{"t":"p","runs":[{"text":"bf16 requires Ampere or newer. Concretely:"}]},{"t":"ul","items":[[{"text":"Has bf16","b":true},{"text":": A100, H100, L4, L40S, A10G, RTX 30-series and 40-series, and newer."}],[{"text":"Does not","b":true},{"text":": the "},{"text":"T4","b":true},{"text":", which is Turing. And the T4 is exactly what free Google Colab and Kaggle give you."}]]},{"t":"p","runs":[{"text":"So on a free tier you are on fp16, with loss scaling, and a divergence at 2e-4 that would have been fine on an A100 is a real possibility. The mitigations on a T4 are: keep the learning rate at the low end, do not skip warmup, and if you see repeated scaler reductions, drop the learning rate rather than investigating anything cleverer."}]},{"t":"p","runs":[{"text":"Check it in one line before you configure anything:"}]},{"t":"code","lang":"python","label":"Python","text":"import torch\nprint(torch.cuda.get_device_name(0), torch.cuda.is_bf16_supported())"},{"t":"h2","id":"mixed-precision-means-mixed","runs":[{"text":"Mixed precision means mixed"}]},{"t":"p","runs":[{"text":"“bf16 training” does not mean everything is bf16. The standard arrangement keeps an "},{"text":"fp32 master copy","b":true},{"text":" of the weights and does the optimiser update in fp32, while forward and backward run in bf16. The reason is accumulation: adding a tiny update to a large weight in bf16 loses the update entirely, because the result rounds back to the original value."}]},{"t":"p","runs":[{"text":"This is also where the 16-bytes-per-parameter figure for full fine-tuning comes from: 4 bytes of fp32 master weights, 4 of fp32 gradients, 8 of AdamW state, with the bf16 working copies on top."}]},{"t":"p","runs":[{"text":"For QLoRA the arrangement differs again: base weights are 4-bit, the compute dtype for matrix multiplications is bf16, and only the small adapter is kept and updated in higher precision. That is three precisions in one model, which is normal and not a sign of anything wrong."}]},{"t":"h2","id":"fp8-and-lower","runs":[{"text":"fp8 and lower"}]},{"t":"p","runs":[{"text":"H100 and newer support fp8 for training, with per-tensor scaling to manage the tiny range. It is real and it is used at scale, but the tooling is still specialist and the gains show up on large pretraining runs rather than on a LoRA over 3,000 examples. For the work this course describes, bf16 where available and fp16 where not is the whole decision."}]},{"t":"p","runs":[{"text":"One caveat worth keeping: a model "},{"text":"trained","i":true},{"text":" in bf16 and then "},{"text":"served","i":true},{"text":" in fp16 can behave slightly differently, because a value near the edge of fp16’s range that was fine during training now saturates. It is rare, and when it happens it looks like a serving bug rather than a precision bug, which is why it takes so long to find."}]}],"check":{"question":[{"text":"A fine-tune on a free Colab T4 runs fine for 200 steps, then the logs fill with the gradient scaler halving its scale, and progress stalls. What is the mechanism?"}],"options":[{"text":[{"text":"The T4 has no bf16, so fp16 loss scaling is in use and overflows are skipping optimiser steps."}],"correct":true,"tempting":null},{"text":[{"text":"The T4’s 16 GB of memory has filled, and the scaler is reducing precision to free space."}],"correct":false,"tempting":[{"text":"Conflates the loss scaler with a memory-management mechanism; the scaler adjusts a multiplier on the loss and frees nothing."}]},{"text":[{"text":"The cosine schedule has decayed the learning rate far enough that gradients now underflow in fp16."}],"correct":false,"tempting":[{"text":"Inverts the cause: the scaler halves its scale in response to overflow to infinity, not underflow, and lower learning rates do not shrink gradients."}]},{"text":[{"text":"bf16 was requested but silently fell back to fp32, so memory pressure is forcing the framework to rescale."}],"correct":false,"tempting":[{"text":"Describes a fallback that would raise memory use without involving the gradient scaler at all, which only exists on the fp16 path."}]}],"order":[1,0,3,2],"why":null},"prev":{"slug":"batch-size-and-accumulation","title":"Batch size, accumulation and the token count","minutes":9},"next":{"slug":"where-the-memory-goes","title":"Where the memory actually goes","minutes":10}}}