Which model, which prompt, which version

The complaint you cannot investigate

A user writes: last Tuesday your assistant told me something wrong, here is a screenshot. You open the code. The prompt has been edited twice since. The provider has silently moved their model pointer to a newer snapshot. Somebody changed the temperature. The retrieval index was rebuilt on Wednesday.

You cannot reproduce the answer, so you cannot say whether it was fixed, and you cannot tell whether the change you are about to make would have prevented it. The deploy lesson gave you a commit SHA for code. A model system needs the same discipline over four more things that all change independently.

The four versions to record with every response

1. Model identity, pinned. Not gpt-latest or claude-latest or whatever the alias is called this year, but the dated snapshot. Aliases move under you, and when they move, quality changes without a deploy. Pin the version, and treat a provider’s deprecation notice as a scheduled piece of work rather than a surprise.
2. Prompt version. The prompt is code. It belongs in the repository, in a file, with a hash, not in a database row somebody edits in an admin panel, and certainly not pasted in three places.
3. Decoding parameters. Temperature, top-p, max tokens, stop sequences, tools available. A temperature of 0.9 makes reproduction impossible in principle; recording the seed where a provider supports one makes it possible again.
4. Retrieval state. If context is fetched, record the index version and the identifiers of the documents that were actually retrieved. “The model hallucinated” and “the retriever returned the wrong document” look identical from outside and require completely different fixes.

Store all of it on the row with the response, or in a log line with the request ID:

{"request_id":"c9f2a1","model":"provider/model-2026-04-12","prompt":"answer_v7:9f31",
 "temp":0.2,"max_tokens":600,"index":"kb-2026-08-30","docs":["kb/412","kb/77"],
 "in_tokens":1204,"out_tokens":388,"latency_ms":2140}

That single line answers the complaint, feeds the cost breakdown from the previous lesson, and tells you which release a regression started in.

A registry, for models you train yourself

If you produce your own models, the artifact is a file, and a file with no provenance is a liability. What to record, at minimum: the training data snapshot or its hash, the code commit, the hyperparameters, the evaluation numbers on the held-out set, who trained it and when.

MLflow is the free, self-hostable standard for this, and its model registry gives you a version number, a stage, and the run that produced it. DVC handles the data-and-artifact half in git. If both feel heavy, a directory named by date and a model.json beside every artifact carrying those six fields is a legitimate starting point: the point is that the fields exist, not that a platform holds them.

One rule: the artifact you evaluated must be the artifact you deploy. Re-training “the same model” for production, with the same script and different randomness, produces a different model with unverified numbers. This is the build-once-run-anywhere argument, transferred.

Evaluate before you ship, on a set that never moves

This course is not the evaluation course, and the details of building a held-out set belong there. The shipping-side rule is short: a model or prompt change is a deploy, so it needs a check that runs automatically and can block it.

In practice that is a file of 50 to 300 examples with expected properties, run in CI, reporting a number. Not a perfect measure (nobody has one) but a tripwire. Teams without it discover regressions from users; teams with it discover them in the pull request. The gap between those two is usually one afternoon of work.

Record the number against the prompt hash so you have a history. A quality figure that only exists in somebody’s memory of last month is not a baseline.

Prompts in the database

There is a real argument for editable prompts: non-engineers can improve them without a deploy. The cost is that your production behaviour is no longer in version control, and the reproduction problem above returns in full.

The compromise that works: prompts live in the repository as the source of truth, an interface lets people propose edits, and an edit becomes a pull request. If you must allow live editing, version every change, record the version with every response, and keep an audit trail of who changed what. Never allow a live edit that leaves no trace.

The smallest version of this lesson

One table, one row per model call, six columns: request ID, model, prompt version, parameters, retrieved document IDs, token counts. It costs an hour. Without it, every question anybody asks about your model’s behaviour is answered with a shrug.