The model you tested is not the model you deployed

Aliases move

Most providers offer two ways to name a model: a friendly alias, and a pinned identifier with a date or version in it. The alias is convenient and it points somewhere different over time. One morning your prompt is being served by a different model, your outputs change, nothing in your repository changed, and the git log is no help at all.

Pin the identifier. Then do the thing that makes the pin useful: log the model id the API returns in the response, not the one you sent. They can differ (an alias resolves, a deprecated version redirects) and the returned value is the only record of what actually ran.

Pinning is a deferral, not an escape. Pinned versions get deprecated, typically with a few months’ notice rather than years. Put the retirement date in a shared calendar with a fortnight’s warning the moment you pin, because the alternative is discovering it from a 404 in production.

Treat a model change as a dependency upgrade

Your prompt is fitted to a particular model. Between versions, any of these can move: how readily it calls tools, how verbose it is, how it formats lists, when it refuses, how it handles a long transcript, and occasionally the tokeniser, which shifts your costs and your truncation points.

So a model change is a major-version bump of your most important dependency. It gets the same treatment: run the golden task set on both, compare per task rather than in aggregate, read the diffs of tasks that changed, then canary.

The tasks that regress are usually specific and fixable: an instruction the old model inferred and the new one wants stated. Twenty minutes of reading beats a week of arguing about whether the new model is better.

Version everything as one thing

Three things determine behaviour: the model id, the prompt version, and the tool schema version. Combine them into a single config version and log it on every run.

Without this, attribution is impossible. Somebody edited a tool description on Tuesday, somebody else changed the model on Wednesday, quality fell on Thursday, and there is no way to say which. With it, you filter runs by config version and the comparison takes a minute.

Keep the previous config deployed rather than deleted, so rollback is changing a value rather than finding an old file.

Reproducibility

Temperature zero and a fixed seed reduce variation. They do not guarantee identical outputs. Providers batch requests together, run on varied hardware, and update inference kernels, and floating-point non-associativity means the same logits are not reproduced bit for bit. Most providers document this as best-effort rather than a guarantee.

So do not build anything that depends on identical outputs from the same input: a cache keyed on exact response text, a test asserting a string match, a diff-based check that assumes stability. Assert on properties: valid schema, correct field values, the right tool called. Those hold; byte equality does not, and a suite built on it will fail intermittently for reasons nobody can reproduce.

Fallback chains that have never been tested are not fallbacks

Chaining providers so a second is tried when the first fails is sensible, and it is more work than adding an except block. Each provider needs its own prompt shaping, its own tool-calling format, its own token limits and its own quirks around structured output. A chain assembled on the assumption that the interfaces are interchangeable produces a fallback that fails the moment it is needed.

Two habits make it real. Run the golden set against every provider in the chain, not just the primary, and keep the results. And exercise the fallback path deliberately on a schedule (force a failover once a month during working hours) because an untested recovery path degrades silently while the primary keeps working.

Say what you ran

When somebody asks why an output looks different from last month, the answer should be a query: this run used config version 14, that one used version 11, here are the three things that differ. That capability costs one extra column and it is the difference between a technical answer and a shrug, which in a regulated setting is the difference between a finding and a conversation.