Versioning so a model can be explained later

The question you will be asked

Someone forwards a complaint about a decision the system made in March. What produced that answer?

Answering it needs six things linked together: the request, the model version that served it, the adapter artefact, the base model revision, the dataset that trained it, and the configuration. Miss any one and the chain breaks.

The manifest

One file, written at build time, travelling with the artefact:

JSON

{"name": "triage", "version": "3.2.0", "built": "2026-09-15T11:04Z",
 "base_model": "meta-llama/Llama-3.1-8B-Instruct",
 "base_revision": "0e9e39f31a2fc8d",
 "dataset": "ticket-triage-sft@v3",
 "dataset_sha256": "9f2c1e…",
 "config_sha256": "71ba04…",
 "eval_set": "eval-v3", "eval_sha256": "c31d77…",
 "git_commit": "4f1a09",
 "results": {"task": 0.918, "canary": 0.812, "refusal": 0.94},
 "quantisation": "Q4_K_M",
 "seed": 0}

Everything in it is a hash or an identifier that resolves to exact bytes. “We trained on the September dataset” is a memory; 9f2c1e… is a fact.

Log the version on every request

JSON

{"ts": "2026-03-04T09:12Z", "request_id": "r_88412",
 "model": "triage@3.2.0", "adapter": "triage-3.2.0", "latency_ms": 340}

This single field is what connects a complaint to an artefact. Without it, a March request maps to “whatever was deployed in March”, which nobody is certain about after two rollbacks.

Version numbers that carry meaning

A convention worth adopting:

Major: the base model changed. Behaviour may differ everywhere; re-evaluate everything.
Minor: the dataset changed. Same base, new training data.
Patch: the same weights, a different build: a new quantisation, a new serving format.

So 3.2.0 and 3.2.1 are the same model in different formats, and their evaluation numbers will differ slightly, which is exactly the sort of thing that looks like a mystery until the version scheme tells you what changed.

Where the bytes live

All free:

Code and configs: git.
Datasets: a Hugging Face dataset repository (private repositories are free), or DVC or git-lfs against your own storage. All three give content-addressed versions.
Model artefacts: a Hub model repository, or object storage with the version in the path.
Run metadata and results: MLflow’s registry if you want a searchable index, or a JSON file per build if you do not.

A fully adequate small-team setup is: code in git, dataset and adapter in private Hub repositories, manifest committed to git beside the config. No infrastructure, no cost, and it answers the March question.

Retention

Keep enough to explain a decision for as long as you might be asked about it. That period is set by your product and your obligations, not by disk cost; and a LoRA adapter is a few hundred megabytes, so the disk cost is not the constraint.

What to keep per version: the manifest, the adapter, the evaluation results, and a pointer to the dataset version. What you do not need to keep: every intermediate checkpoint, and the optimiser states, which are large and only useful for resuming a run you will not resume.

Three things that break this in practice

Every versioning scheme that fails does so in one of three ways, and all three are cheap to prevent.

A configuration value that is not in the hash. The config file is hashed, but the batch size comes from an environment variable, or the base model path from a shell default. Two runs with identical config hashes then differ. The fix is to resolve every value into the config file before hashing it, and to have the training script write out the resolved config as an artefact rather than the template.

A dataset built by something that is not deterministic. A notebook cell that samples without a seed, a query with LIMIT 20000 and no ORDER BY, a filter that reads the current date. The hash then records what you built, which is correct and useless, because rerunning the builder produces a different file. Seed everything in the pipeline and pin the query, or accept that the dataset is an artefact to be stored rather than a recipe to be rerun: both work, and pretending you have the second when you have the first does not.

A floating reference to the base model. main, latest, or a tag that was moved. The manifest looks complete and resolves to different bytes over time. Always record the commit hash of the model repository, not its name.

Promotion, not just versioning

A version number says what something is. A stage says where it is allowed to run.

Text

triage@3.2.0   stage: production   since 2026-09-15
triage@3.3.0   stage: staging      shadowing production traffic
triage@3.4.0   stage: development

Three stages is enough. The value is that “which model is in production” becomes a fact stored somewhere rather than an inference from deployment logs, and rolling back becomes moving a pointer rather than rebuilding something.

The test

Once a quarter, pick a version at random and rebuild it from its manifest. Compare its evaluation numbers against the recorded ones.

If they match within noise, your versioning works. If you cannot rebuild it at all, you have discovered that on a Tuesday of your choosing rather than during an incident, which is the entire value of the exercise.

Every lesson here is open. Your progress is recorded once you pass the Running Models Yourself exam.

Contents