FP8 and models born quantised

Integers versus floats at low precision

Everything so far quantised to integers: a scale, an offset, and a small whole number. Floating-point formats spend their bits differently (some on an exponent, some on a mantissa) and that changes what they handle well.

An 8-bit float in the E4M3 layout has a sign bit, four exponent bits and three mantissa bits. Compared with an 8-bit integer, it has far less precision near any given magnitude and a far wider range. For neural network weights and activations, which have long-tailed distributions with rare large outliers, range turns out to matter more than uniform precision. An outlier that would stretch an integer block’s scale and coarsen every weight beside it lands in a higher exponent.

That is why FP8 quantisation is close to lossless in practice while 8-bit integer quantisation of activations has always been fiddly.

What FP8 needs and what it gives

Hardware support arrived with NVIDIA’s Ada and Hopper generations and is present on newer AMD parts. On those cards the tensor cores execute FP8 natively, so you get both halved memory and roughly doubled arithmetic throughput against fp16.

vllm serve <model> --quantization fp8 --kv-cache-dtype fp8

Two things make this attractive beyond the numbers. It generally needs no calibration data: a simple per-tensor or per-channel scale is enough, so you can quantise on the fly at load time. And it applies well to the KV cache, where FP8’s range handles the outliers that make 4-bit integer caches degrade long-context recall.

The honest limitation: on a card without FP8 hardware, the format is emulated and you lose the speed benefit while keeping the memory one. Check your card’s generation before assuming the flag helps.

Four-bit floats

NVIDIA’s Blackwell generation adds NVFP4, a 4-bit float with a two-level scaling scheme, and the OCP MXFP4 format shares the idea. The claim is 4-bit memory with quality close to FP8, and early evidence supports it for large models. Some model families have begun shipping natively in MXFP4 rather than being quantised afterwards.

Treat the quality claims with the usual caution (they are made by the people selling the hardware, measured on benchmarks they chose) but the direction is clear and the arithmetic is sound.

Quantisation-aware training, and models born small

There is a distinction worth holding onto. Everything above is post-training quantisation: take a finished model, compress it, accept some loss.

Quantisation-aware training simulates the rounding during training, so the model learns weights that survive it. The result at 4 bits is meaningfully better than post-training quantisation of the same model, because the network had the chance to route around the precision it was going to lose. Several recent releases ship QAT variants specifically for local use, and where one exists it is strictly the better download at the same bit rate.

Further along, some models are trained in low precision from the start. Ternary and 1.58-bit architectures, where each weight is one of three values, are a genuine research direction with working implementations. They are not drop-in quantisations of existing models (the architecture and training differ) and current results at useful sizes remain behind conventional models. Worth watching, not yet worth planning around.

What to actually do

The decision is short:

• Consumer card, older than Ada, model must be small: 4-bit integer quantisation, GGUF or AWQ. FP8 buys you memory and no speed.
• Ada, Hopper, Blackwell or a recent AMD datacentre part, and the model fits at 8 bits: FP8 for weights and cache. Nearly lossless, no calibration, fast.
• A QAT variant exists at your target bit rate: take it over any post-training quant of the same model.
• A model published natively in a 4-bit float format: use it as published rather than re-quantising it; the authors trained for that representation.

The thing to remember about precision generally

Every format above is a claim that some information in the weights was not needed. Whether that is true depends on the task, which is why the same quant can be invisible on summarising and clearly damaging on code or on a low-resource language. No format removes that. The formats differ in how gracefully they fail and in which capability goes first, and the only way to know which applies to you is the measurement in the next lesson.