Being able to build the same model twice

The test

Can you rebuild, byte for byte, the model that is running in production right now?

Most teams cannot. That matters for three practical reasons: you cannot debug a model you cannot reconstruct, you cannot demonstrate to an auditor what a decision was based on, and you cannot tell whether a change you made improved things or whether something else moved underneath you.

The four things that must be pinned

1. Code. A git commit hash. The easy one, and the one everybody does.

2. Data. The hard one. “The customers table” is not a version; it changes every night. You need either a snapshot, or a query with an explicit as-of timestamp, or a content hash of the exact extract. DVC and lakeFS are free tools for this; a dated Parquet file in object storage with its hash recorded is a perfectly good version-one answer.

3. Environment. Library versions. pip freeze > requirements.txt at minimum; a container image digest if you can. scikit-learn changes default parameters between versions, and a model trained under 1.2 and retrained under 1.6 with identical code can differ meaningfully.

4. Randomness. Every seed, everywhere.

import random, numpy as np, torch
SEED = 0
random.seed(SEED); np.random.seed(SEED); torch.manual_seed(SEED)

And pass random_state=SEED to every splitter, sampler and estimator that accepts it. A model whose score moves by 1.5 points between runs cannot be compared against anything.

Where seeds are not enough

Worth knowing so you do not chase a ghost.

GPU non-determinism. Some CUDA operations accumulate floating-point sums in a non-deterministic order, so results differ in the last decimal places and can amplify over a long training run. torch.use_deterministic_algorithms(True) forces deterministic implementations where they exist and is measurably slower.

Parallelism. Multi-threaded reductions sum in whatever order threads finish. Same cause, same tiny differences.

Hardware. Different CPU instruction sets can produce slightly different floating-point results for the same code.

So exact bitwise reproducibility across machines is often not achievable. Statistical reproducibility is: the same data, code, environment and seeds should give a score within noise of the original, and if it does not, something is unpinned. That is the standard to hold yourself to.

Tracking runs

Once you are running dozens of experiments, memory and file names stop working.

import mlflow
with mlflow.start_run():
    mlflow.log_params({"max_depth": 6, "lr": 0.05, "data": "2026-08-14"})
    mlflow.log_metrics({"val_auc": 0.847, "val_ap": 0.312})
    mlflow.log_artifact("model.joblib")

MLflow is free and runs locally with no server. Weights and Biases has a free tier and a better interface. Or a CSV appended to by your training script, which is adequate for one person and infinitely better than nothing.

What to log, in every run: the data version, the code commit, the seed, every hyperparameter, every metric on every slice, the training duration, and the model artefact. When someone asks in six months why the model changed, this is the only thing that answers.

The model registry

A place where model artefacts live with metadata: version, training data, metrics, stage (staging, production, archived), and who approved the promotion.

This is what makes “which model is in production and what was it trained on” a lookup rather than an investigation, and it is what makes rollback a pointer change. MLflow includes one; a directory with a naming convention and a JSON file per model is a legitimate starting point.

The lightweight version that actually gets done

If all of the above sounds like a quarter of work, here is the version that takes an afternoon and gets you 80% of the benefit:

{
  "model_version": "2026-09-06-a",
  "git_commit": "4294d8f",
  "data_snapshot": "s3://bucket/train/2026-09-01.parquet",
  "data_sha256": "e3b0c442...",
  "seed": 0,
  "sklearn": "1.6.1",
  "metrics": {"val_auc": 0.847, "test_auc": 0.839},
  "threshold": 0.62,
  "trained_by": "pipeline",
  "trained_at": "2026-09-06T11:04:00Z"
}

One JSON file written next to every model artefact by the training script. That is it. It answers almost every question anyone will ask, and the discipline of writing it forces you to actually know the data version, which, more often than anyone admits, is the moment a team discovers they do not.

Reproducibility is what makes comparison possible

One closing argument, because reproducibility is usually presented as a compliance chore.

Every claim in modules 4 and 5 (this model beats that one, this feature helped, this threshold is right) assumes that rerunning the same configuration gives the same answer. When it does not, you cannot tell an improvement from a different random seed, and every comparison you make becomes an opinion supported by a number.

So the discipline is not paperwork. It is the precondition for the measurement being about the model at all. A team with unpinned data is not doing worse science slowly; it is doing something that only resembles science, and it usually finds out during the first serious disagreement about whether a change helped.