Reproducibility, and the noise floor

Bit-identical is harder than it looks

from transformers import set_seed
set_seed(0)     # python, numpy, torch, cuda

That covers weight initialisation, dropout, data shuffling and sampling. It does not give you a bit-identical run, because GPU reductions are not deterministic by default: floating-point addition is not associative, and the order in which thousands of parallel threads accumulate a sum varies between executions.

Full determinism is available and it is expensive:

import torch, os
os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"
torch.use_deterministic_algorithms(True)
torch.backends.cudnn.benchmark = False

This costs throughput, and some operations have no deterministic implementation at all: the call will raise rather than silently be non-deterministic, which is the right behaviour and is also how you discover that your model uses one of them.

And even with all of it, results differ across GPU models, driver versions and library versions. Reproducibility across machines is not achievable by flags.

Reproducible enough

The practical goal is not bit-identity. It is: somebody with this repository can produce a model that performs the same within noise. That needs a record, not a flag.

{"git_commit": "4f1a09",
 "config_sha256": "71ba04…",
 "dataset_sha256": "9f2c1e…",
 "base_model": "meta-llama/Llama-3.1-8B-Instruct",
 "base_revision": "0e9e39f",
 "seed": 0,
 "gpu": "1×RTX 4090",
 "versions": {"torch": "…", "transformers": "…", "trl": "…", "peft": "…"}}

The two hashes are the ones people omit and the two that matter most. “We trained on the customer dataset” is not a reproducible statement; a sha256 of the exact JSONL is.

Pin the base model revision too. Model repositories are edited (tokenizer configs get updated, chat templates get fixed) and main six months from now is a different artefact.

The measurement everybody skips

Here is the part that changes how you read every result in this course.

Run the same configuration three times with seeds 0, 1 and 2. Evaluate all three on your held-out set. The spread between them is your noise floor.

seed 0: 0.871
seed 1: 0.858
seed 2: 0.884
→ range 2.6 points, sd ≈ 1.3

Now look at the last four “improvements” you recorded. If they were one or two points, you measured nothing. You measured seed variance and attributed it to a decision.

This is the single most common error in applied fine-tuning, and three extra runs (a few hours and a few pounds) is the whole cost of not making it. Until you know your noise floor, you cannot distinguish a real effect from a coin flip, and every ablation you run is uninterpretable.

Once you have it, the rule is simple: an improvement smaller than about twice your seed standard deviation is not an improvement. Either accept that, or run each configuration three times and compare the means, which is what the difference is worth measuring, if it is worth measuring.

Where the variance comes from

Worth knowing, because it tells you how to reduce it:

• Data order dominates on small datasets. With 800 examples, the sequence in which the model sees them changes the result materially.
• Adapter initialisation: the random A matrix.
• Dropout masks.
• Non-deterministic reductions, the smallest contributor.

More data narrows the first and largest source, which is one more reason the dataset is the lever that matters.

Reporting a result with its noise

Once you have the floor, the honest form of a result is a mean and a range, not a single figure:

baseline   0.871 ± 1.3  (3 seeds)
+ replay   0.870 ± 1.1  (3 seeds)   task unchanged
           canary 0.62 → 0.81       the actual finding

Note what that reporting does. It makes the task number a non-finding explicitly, and puts the canary movement (which is well outside the noise) where the reader’s attention belongs. A single-figure table would have invited an argument about 0.871 versus 0.870.

The evaluation must be fixed

Seed your evaluation too. If you generate at temperature 0.7 for evaluation, the same model scores differently on consecutive runs and you have added a second source of noise on top of the training one. Evaluate greedily, or with a fixed seed and enough samples to average, and record which you did.

A result without a stated evaluation procedure is not a result. A result without a noise floor is not a comparison.