Random numbers that are not, and what a seed does and does not fix

Determinism wearing a disguise

A computer cannot produce a random number. What it produces is a pseudo-random sequence: a fixed recipe that turns one state into the next, chosen so that the output passes every statistical test for randomness while being entirely determined by where it started. The starting state is the seed. Same seed, same recipe, same sequence, forever.

The simplest recipe, a linear congruential generator, is one line: state = (a × state + c) mod m. Modern generators, the Mersenne Twister behind older NumPy and Python's random, and the PCG64 behind np.random.default_rng, are more elaborate but no less deterministic. Their period, the number of outputs before the sequence repeats, is around 2^19937 for the Twister and 2^128 for PCG64. Neither will repeat in your lifetime. But a sequence that never repeats is not the same as one that cannot be predicted; if the seed is known, every number is known.

What a seed buys

Set it, and a shuffle, a dropout mask, an initialisation and a train-test split are the same on every run. That is what makes a bug reproducible and a comparison fair. Set it in every library that draws numbers, because each keeps its own state:

import random, numpy as np, torch
random.seed(0)
np.random.seed(0)                       # legacy global state
rng = np.random.default_rng(0)          # preferred: an explicit generator you pass around
torch.manual_seed(0)                    # CPU and all GPUs

The explicit generator is better than the global seed, because two parts of a program drawing from one global stream change each other's numbers when one of them adds a draw. Give each component its own generator, seeded from a master seed, and adding a draw in one place no longer changes the shuffle in another.

What a seed does not buy

Bitwise identical training on a GPU. The previous lessons showed that floating-point addition depends on order and that a GPU adds in whatever order its threads arrive. The seed fixes the dropout mask and the initial weights; it does not fix the reduction order. Two runs with the same seed diverge in their last bits at the first step and in their third digit within a few thousand. torch.use_deterministic_algorithms(True) forces fixed-order kernels, at a cost in speed and with some operations refusing to run; it is the price of exact reproducibility, and it is sometimes worth paying.

Identical results across machines. Different hardware, driver versions and library builds choose different kernels with different orders. Reproducibility across machines is a matter of agreeing to three digits, not thirty-two bits.

A stable split when the data grow. Shuffling with a seed and taking the first 80 per cent as training is reproducible only until a row is added, after which every row's position shifts and yesterday's test examples leak into today's training set. The fix uses module 7's hashing: assign each row by a hash of its id, hash(id) mod 100 < 80, so that a row's split depends on the row alone. New rows land in either split; old rows never move.

Python's string hashing. It is randomised per process by design, which changes the order of iteration over a set of strings from one run to the next. A vocabulary built by iterating a set is different on every run unless PYTHONHASHSEED is fixed or the set is sorted first. This is the single most common cause of "the same code produces a different model".

Turning uniform into anything

A generator produces uniform numbers between 0 and 1. Every other distribution is manufactured from those. The general method is the inverse cumulative distribution: if F is the cumulative distribution you want, then F^(−1)(u) for uniform u has that distribution. For the exponential with rate λ:

x = −ln(u) / λ

For the normal, the inverse has no closed form, and the Box-Muller transform takes two uniforms and returns two independent normals:

z₁ = √(−2 ln u₁) × cos(2π u₂)
z₂ = √(−2 ln u₁) × sin(2π u₂)

For a discrete distribution, a token sampled from a softmax, the method is the same: compute the cumulative sums, draw u, and pick the first token whose cumulative probability exceeds u. That is what torch.multinomial does, and it is why sampling a token costs a scan over the vocabulary rather than a lookup.

Randomness you did not ask for

DataLoader workers each get a seed derived from the main one plus their worker id; if you seed NumPy inside a worker without that id, all workers produce the same augmentations. Dropout at evaluation time, if the model was never switched out of training mode, makes the same input give different outputs. Sampling with temperature > 0 is random by intent. And any code that reads the clock, a network, or the filesystem order has a source of variation no seed touches.

The habit

Seed everything, pass generators explicitly, hash for splits, sort before assigning ids, and expect two GPU runs to agree to three digits rather than exactly. When they agree to fewer, look for the nondeterminism, and when they agree exactly, check that the second run actually ran.