Swapping the model under a live product

What a model swap actually changes

Everything downstream of the weights. A different model has a different chat template, different sampling defaults, different refusal thresholds, different verbosity, a different tokenizer with a different cost per request, different tool-call formatting, and different behaviour on the prompt you tuned for its predecessor.

The mistake is to treat it as a configuration change. It is a release, and it needs the same discipline as one.

The frozen prompt problem

Your prompt was tuned against the old model. It contains phrasings, an ordering and an example set that worked there. On a new model, some of it helps and some of it hurts, and there is no way to know which without measuring.

Teams routinely evaluate a new model with the old model’s prompt, conclude it is worse, and stay put. Sometimes that is correct. Often the new model with a prompt tuned for it beats both.

So the honest comparison is best-of-A against best-of-B: an hour of prompt tuning on each, on the development split, then compare on the held-out split. Anything less is comparing a tuned system against an untuned one.

The checklist

Before any swap reaches users:

1. Held-out task metric, both models, identical settings, with the noise floor stated.
2. Regression set: the fifty items from the fine-tuning block, covering general behaviour, other languages, other formats and refusals.
3. False-refusal set, because refusal thresholds vary widely between families and this changes silently.
4. Cost per task, measured, not estimated. The tokenizer changed, so the input token count changed.
5. Latency at real concurrency, p95, TTFT separately.
6. Format and tool-call compatibility, tested against the actual serving stack rather than assumed.
7. Licence, re-checked. The new model may have obligations the old one did not.

Seven checks, most of them scripts you already have from earlier lessons. A morning.

Rolling it out

Shadow first. Send a copy of live traffic to the new model, discard its responses, and store both outputs. After a day you have thousands of paired comparisons on real inputs, at no risk. Sample two hundred and read them. This is the highest-value step in the list and the most often skipped.

Then canary. 1% of traffic, then 10%, then 50%. Watch the metrics that would show harm: error rate, retry rate, refusal rate, p95 latency, cost per task, and whatever product signal you have: resolution rate, escalation rate, thumbs-down.

Define the stop condition in advance. A specific metric crossing a specific threshold rolls the change back automatically or wakes somebody. Deciding what counts as bad while looking at a graph at 2am is how bad changes stay.

Keep the old configuration deployable. Both models, both prompts, both sampler settings, selectable by configuration. Reverting should be a deploy, not a re-tune.

The swap you did not make

The earlier lesson on providers described this: an aggregator routing to a cheaper provider, or a provider changing its serving precision or context cap, is a model swap performed on your product without your knowledge.

Two defences. Pin the provider and the quantisation where the platform allows it. And run a small canary evaluation on a schedule: twenty items, daily, comparing against recorded expected outputs. It costs pennies and it turns a mysterious quality complaint three weeks later into an alert on the day.

The same applies to self-hosting with an unpinned revision. main moves. A pinned commit hash is the difference between a deployment you can reason about and one that changes when nobody deployed anything.

Write down why

When the swap is done, record the decision: the two numbers, the date, the revisions, the prompt commits, and one sentence about why the new model won. In six months somebody will ask why you are on this model, and the alternative to a written answer is repeating the whole exercise from memory.