{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"fp8-and-native-low-precision","title":"FP8 and models born quantised","url":"https://addaly.com/learn/running-models-yourself/fp8-and-native-low-precision","position":23,"of":82,"minutes":8,"answer":"FP8 spends its bits on exponent rather than precision, which is why it survives the outliers that hurt integer quantisation, and a quantisation-aware-trained model at the same bit rate beats any post-training quant of it.","keyIdea":"FP8 spends its bits on exponent rather than precision, which is why it survives the outliers that hurt integer quantisation, and a quantisation-aware-trained model at the same bit rate beats any post-training quant of it."},"body":{"format":"markdown","text":"## Integers versus floats at low precision\n\nEverything so far quantised to integers: a scale, an offset, and a small whole number. Floating-point formats spend their bits differently (some on an exponent, some on a mantissa) and that changes what they handle well.\n\nAn 8-bit float in the E4M3 layout has a sign bit, four exponent bits and three mantissa bits. Compared with an 8-bit integer, it has far less precision near any given magnitude and a far wider range. For neural network weights and activations, which have long-tailed distributions with rare large outliers, range turns out to matter more than uniform precision. An outlier that would stretch an integer block's scale and coarsen every weight beside it lands in a higher exponent.\n\nThat is why FP8 quantisation is close to lossless in practice while 8-bit integer quantisation of *activations* has always been fiddly.\n\n## What FP8 needs and what it gives\n\nHardware support arrived with NVIDIA's Ada and Hopper generations and is present on newer AMD parts. On those cards the tensor cores execute FP8 natively, so you get both halved memory and roughly doubled arithmetic throughput against fp16.\n\n```bash\nvllm serve <model> --quantization fp8 --kv-cache-dtype fp8\n```\n\nTwo things make this attractive beyond the numbers. It generally needs **no calibration data**: a simple per-tensor or per-channel scale is enough, so you can quantise on the fly at load time. And it applies well to the KV cache, where FP8's range handles the outliers that make 4-bit integer caches degrade long-context recall.\n\nThe honest limitation: on a card without FP8 hardware, the format is emulated and you lose the speed benefit while keeping the memory one. Check your card's generation before assuming the flag helps.\n\n## Four-bit floats\n\nNVIDIA's Blackwell generation adds NVFP4, a 4-bit float with a two-level scaling scheme, and the OCP MXFP4 format shares the idea. The claim is 4-bit memory with quality close to FP8, and early evidence supports it for large models. Some model families have begun shipping natively in MXFP4 rather than being quantised afterwards.\n\nTreat the quality claims with the usual caution (they are made by the people selling the hardware, measured on benchmarks they chose) but the direction is clear and the arithmetic is sound.\n\n## Quantisation-aware training, and models born small\n\nThere is a distinction worth holding onto. Everything above is **post-training quantisation**: take a finished model, compress it, accept some loss.\n\n**Quantisation-aware training** simulates the rounding during training, so the model learns weights that survive it. The result at 4 bits is meaningfully better than post-training quantisation of the same model, because the network had the chance to route around the precision it was going to lose. Several recent releases ship QAT variants specifically for local use, and where one exists it is strictly the better download at the same bit rate.\n\nFurther along, some models are trained in low precision from the start. Ternary and 1.58-bit architectures, where each weight is one of three values, are a genuine research direction with working implementations. They are not drop-in quantisations of existing models (the architecture and training differ) and current results at useful sizes remain behind conventional models. Worth watching, not yet worth planning around.\n\n## What to actually do\n\nThe decision is short:\n\n- **Consumer card, older than Ada, model must be small**: 4-bit integer quantisation, GGUF or AWQ. FP8 buys you memory and no speed.\n- **Ada, Hopper, Blackwell or a recent AMD datacentre part, and the model fits at 8 bits**: FP8 for weights and cache. Nearly lossless, no calibration, fast.\n- **A QAT variant exists at your target bit rate**: take it over any post-training quant of the same model.\n- **A model published natively in a 4-bit float format**: use it as published rather than re-quantising it; the authors trained for that representation.\n\n## The thing to remember about precision generally\n\nEvery format above is a claim that some information in the weights was not needed. Whether that is true depends on the task, which is why the same quant can be invisible on summarising and clearly damaging on code or on a low-resource language. No format removes that. The formats differ in how gracefully they fail and in which capability goes first, and the only way to know which applies to you is the measurement in the next lesson."},"check":{"question":"Two teams quantise the same model to 8 bits: one to INT8, one to FP8 E4M3. The FP8 version holds up noticeably better on long-context recall. What is the mechanism?","options":[{"text":"FP8 has greater dynamic range, so rare large values in the KV cache do not stretch a shared scale and coarsen every neighbouring value the way they do in a block of integers.","correct":true},{"text":"FP8 kernels run on tensor cores, and the higher throughput means less numerical drift accumulates over a long sequence.","correct":false,"tempting":"Attributes an accuracy result to execution speed; kernel throughput does not change the values computed."},{"text":"INT8 stores only positive values, so negative activations must be offset, and the offset error accumulates with sequence length.","correct":false,"tempting":"Misdescribes the format; signed and asymmetric integer schemes both handle negatives without accumulating offset error."},{"text":"FP8 uses more bits per element in practice because of its scale factors, so the comparison is not like for like.","correct":false,"tempting":"Both are eight bits per element; scales exist in both schemes and do not account for the quality difference."}]},"prev":{"slug":"gpu-quantisation-formats","title":"GPTQ, AWQ and the GPU-side formats","url":"https://addaly.com/learn/running-models-yourself/gpu-quantisation-formats"},"next":{"slug":"quantising-a-model-yourself","title":"Quantising a model yourself","url":"https://addaly.com/learn/running-models-yourself/quantising-a-model-yourself"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'FP8 and models born quantised', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/fp8-and-native-low-precision (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"FP8 and models born quantised","courseTitle":"Running Models Yourself","module":{"index":3,"slug":"quantisation-and-formats","title":"Quantisation and model formats"},"answer":null,"keyIdea":"FP8 spends its bits on exponent rather than precision, which is why it survives the outliers that hurt integer quantisation, and a quantisation-aware-trained model at the same bit rate beats any post-training quant of it.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"integers-versus-floats-at-low-precision","runs":[{"text":"Integers versus floats at low precision"}]},{"t":"p","runs":[{"text":"Everything so far quantised to integers: a scale, an offset, and a small whole number. Floating-point formats spend their bits differently (some on an exponent, some on a mantissa) and that changes what they handle well."}]},{"t":"p","runs":[{"text":"An 8-bit float in the E4M3 layout has a sign bit, four exponent bits and three mantissa bits. Compared with an 8-bit integer, it has far less precision near any given magnitude and a far wider range. For neural network weights and activations, which have long-tailed distributions with rare large outliers, range turns out to matter more than uniform precision. An outlier that would stretch an integer block’s scale and coarsen every weight beside it lands in a higher exponent."}]},{"t":"p","runs":[{"text":"That is why FP8 quantisation is close to lossless in practice while 8-bit integer quantisation of "},{"text":"activations","i":true},{"text":" has always been fiddly."}]},{"t":"h2","id":"what-fp8-needs-and-what-it-gives","runs":[{"text":"What FP8 needs and what it gives"}]},{"t":"p","runs":[{"text":"Hardware support arrived with NVIDIA’s Ada and Hopper generations and is present on newer AMD parts. On those cards the tensor cores execute FP8 natively, so you get both halved memory and roughly doubled arithmetic throughput against fp16."}]},{"t":"code","lang":"bash","label":"Bash","text":"vllm serve <model> --quantization fp8 --kv-cache-dtype fp8"},{"t":"p","runs":[{"text":"Two things make this attractive beyond the numbers. It generally needs "},{"text":"no calibration data","b":true},{"text":": a simple per-tensor or per-channel scale is enough, so you can quantise on the fly at load time. And it applies well to the KV cache, where FP8’s range handles the outliers that make 4-bit integer caches degrade long-context recall."}]},{"t":"p","runs":[{"text":"The honest limitation: on a card without FP8 hardware, the format is emulated and you lose the speed benefit while keeping the memory one. Check your card’s generation before assuming the flag helps."}]},{"t":"h2","id":"four-bit-floats","runs":[{"text":"Four-bit floats"}]},{"t":"p","runs":[{"text":"NVIDIA’s Blackwell generation adds NVFP4, a 4-bit float with a two-level scaling scheme, and the OCP MXFP4 format shares the idea. The claim is 4-bit memory with quality close to FP8, and early evidence supports it for large models. Some model families have begun shipping natively in MXFP4 rather than being quantised afterwards."}]},{"t":"p","runs":[{"text":"Treat the quality claims with the usual caution (they are made by the people selling the hardware, measured on benchmarks they chose) but the direction is clear and the arithmetic is sound."}]},{"t":"h2","id":"quantisation-aware-training-and-models-born-small","runs":[{"text":"Quantisation-aware training, and models born small"}]},{"t":"p","runs":[{"text":"There is a distinction worth holding onto. Everything above is "},{"text":"post-training quantisation","b":true},{"text":": take a finished model, compress it, accept some loss."}]},{"t":"p","runs":[{"text":"Quantisation-aware training","b":true},{"text":" simulates the rounding during training, so the model learns weights that survive it. The result at 4 bits is meaningfully better than post-training quantisation of the same model, because the network had the chance to route around the precision it was going to lose. Several recent releases ship QAT variants specifically for local use, and where one exists it is strictly the better download at the same bit rate."}]},{"t":"p","runs":[{"text":"Further along, some models are trained in low precision from the start. Ternary and 1.58-bit architectures, where each weight is one of three values, are a genuine research direction with working implementations. They are not drop-in quantisations of existing models (the architecture and training differ) and current results at useful sizes remain behind conventional models. Worth watching, not yet worth planning around."}]},{"t":"h2","id":"what-to-actually-do","runs":[{"text":"What to actually do"}]},{"t":"p","runs":[{"text":"The decision is short:"}]},{"t":"ul","items":[[{"text":"Consumer card, older than Ada, model must be small","b":true},{"text":": 4-bit integer quantisation, GGUF or AWQ. FP8 buys you memory and no speed."}],[{"text":"Ada, Hopper, Blackwell or a recent AMD datacentre part, and the model fits at 8 bits","b":true},{"text":": FP8 for weights and cache. Nearly lossless, no calibration, fast."}],[{"text":"A QAT variant exists at your target bit rate","b":true},{"text":": take it over any post-training quant of the same model."}],[{"text":"A model published natively in a 4-bit float format","b":true},{"text":": use it as published rather than re-quantising it; the authors trained for that representation."}]]},{"t":"h2","id":"the-thing-to-remember-about-precision-generally","runs":[{"text":"The thing to remember about precision generally"}]},{"t":"p","runs":[{"text":"Every format above is a claim that some information in the weights was not needed. Whether that is true depends on the task, which is why the same quant can be invisible on summarising and clearly damaging on code or on a low-resource language. No format removes that. The formats differ in how gracefully they fail and in which capability goes first, and the only way to know which applies to you is the measurement in the next lesson."}]}],"check":{"question":[{"text":"Two teams quantise the same model to 8 bits: one to INT8, one to FP8 E4M3. The FP8 version holds up noticeably better on long-context recall. What is the mechanism?"}],"options":[{"text":[{"text":"FP8 has greater dynamic range, so rare large values in the KV cache do not stretch a shared scale and coarsen every neighbouring value the way they do in a block of integers."}],"correct":true,"tempting":null},{"text":[{"text":"FP8 kernels run on tensor cores, and the higher throughput means less numerical drift accumulates over a long sequence."}],"correct":false,"tempting":[{"text":"Attributes an accuracy result to execution speed; kernel throughput does not change the values computed."}]},{"text":[{"text":"INT8 stores only positive values, so negative activations must be offset, and the offset error accumulates with sequence length."}],"correct":false,"tempting":[{"text":"Misdescribes the format; signed and asymmetric integer schemes both handle negatives without accumulating offset error."}]},{"text":[{"text":"FP8 uses more bits per element in practice because of its scale factors, so the comparison is not like for like."}],"correct":false,"tempting":[{"text":"Both are eight bits per element; scales exist in both schemes and do not account for the quality difference."}]}],"order":[2,0,3,1],"why":null},"prev":{"slug":"gpu-quantisation-formats","title":"GPTQ, AWQ and the GPU-side formats","minutes":9},"next":{"slug":"quantising-a-model-yourself","title":"Quantising a model yourself","minutes":10}}}