dtypes: what float16 costs, why int8 wraps round, and how to compare floats

Bytes per element

Every array has a dtype, and the dtype decides three things at once: how much memory each element takes, the largest value it can hold, and how many significant figures it keeps.

np.zeros(1_000_000, dtype=np.float64).nbytes   # 8,000,000
np.zeros(1_000_000, dtype=np.float32).nbytes   # 4,000,000
np.zeros(1_000_000, dtype=np.float16).nbytes   # 2,000,000
np.zeros(1_000_000, dtype=np.int8).nbytes      # 1,000,000

A million float64 values is 8 MB. A model with 7 billion parameters stored as float16 is 14 GB; as float32, 28 GB; as int8 after quantisation, 7 GB. That arithmetic — parameters times bytes per parameter — is the single most useful calculation in deciding whether a model fits on a machine, and it is nothing more than nbytes.

Floats: range and precision

• float64: about 15–16 significant decimal figures, largest value around 1.8 × 10³⁰⁸. Python's own float is this. The default for most NumPy operations.
• float32: about 7 significant figures, largest around 3.4 × 10³⁸. The standard for model weights in training and for most GPU arithmetic. Half the memory of float64 and rarely a problem for anything short of accumulating a long sum.
• float16: about 3 significant figures, largest 65,504. Half the memory again. Used for storing weights and for inference on hardware that supports it.
• bfloat16: same 16 bits, but spent differently — the range of float32 with only about 2–3 significant figures. Used in training because the range matters more than the precision there. NumPy does not ship it natively; ml_dtypes adds it.

The practical consequence of float16's ceiling:

x = np.array([12.0], dtype=np.float16)
np.exp(x)         # [inf]  — e^12 ≈ 162,755, above 65,504

A softmax over logits that reach 12 produces inf, and inf / inf is nan, and one nan poisons every later sum. This is why libraries compute softmax by subtracting the maximum logit first — every exponent is then at most e⁰ = 1 — and why you see -inf and nan in a log the day someone stores intermediate values in half precision to save memory. Store in float16 if you must; compute in float32.

Precision has the same shape of consequence in the other direction:

np.float16(1000) + np.float16(0.5)      # 1000.0 — the 0.5 is lost
np.float32(1e8) + np.float32(1)         # 100000000.0 — the 1 is lost

Three significant figures means a value near 1,000 cannot represent a change of 0.5. Summing ten thousand small values into a float16 accumulator loses most of them. NumPy guards against the worst case by accumulating reductions in a wider type, but any arithmetic you write elementwise in float16 has no such guard.

Integers wrap

np.array([127], dtype=np.int8) + 1     # [-128]
np.array([255], dtype=np.uint8) + 1    # [0]

An integer dtype has a fixed range — int8 is −128 to 127, uint8 is 0 to 255, int64 is ±9.2 × 10¹⁸ — and arithmetic that leaves it wraps round without a warning in most configurations. Image pixels are uint8; add brightness to a pixel at 250 and it becomes dark. Token IDs fit in int32; counts of tokens across a large corpus may not fit in int32 and will go negative. Python's own int never does this, which is why the surprise lands on people arriving from plain Python.

np.iinfo(np.int8) and np.finfo(np.float16) print the limits for any type. When a number in an array looks impossible, check them.

Conversion

a = np.array([1.7, 2.2, -0.5])
a.astype(np.int32)          # [1, 2, 0]: truncates toward zero, does not round
np.round(a).astype(int)     # [2, 2, -0]: round first if that is what you meant

astype makes a copy in the new type. Float to int truncates; it does not round. Int to a smaller int wraps. Anything to bool gives False for zero and True otherwise.

Mixed-type arithmetic promotes to the wider type: int32 + float32 is float32, float32 + float64 is float64. A single float64 scalar in a float32 expression can quietly upcast the whole array and double its memory. Check .dtype on the result when memory matters.

Comparing floats

0.1 + 0.2 == 0.3              # False
np.isclose(0.1 + 0.2, 0.3)    # True

This is the same fact as module 1's decimal surprise, in array form. Never compare computed floats with ==. np.isclose(a, b) compares with a tolerance — relative 1e-5 and absolute 1e-8 by default — and np.allclose does it for whole arrays. In tests, pytest.approx or numpy.testing.assert_allclose are the tools. Two embeddings computed on different machines may differ in the last digit; a test that asserts equality is a test that fails on Tuesday.

Choosing

Use float64 for anything small enough not to care and for statistics where accumulated error matters. Use float32 for model inputs, embeddings and anything that will meet a GPU. Use float16 for storage and for inference where the hardware and the library both promise to handle the overflow cases. Use the smallest integer type that certainly holds the range, and check the range rather than assuming it.

Try this now

Compute np.exp of np.arange(0, 20, dtype=np.float16) and find the first inf. Then softmax those same logits in float32 with and without subtracting the max. Finally, make a uint8 array of [250, 251, 252], add 10, and look.