{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"swapping-the-model","title":"Swapping the model under a live product","url":"https://addaly.com/learn/open-models/swapping-the-model","position":70,"of":71,"minutes":9,"answer":"A model swap is a release, not a configuration change: compare each model with a prompt tuned for it, run the regression, refusal, cost and latency checks, shadow real traffic before canarying.","keyIdea":"A model swap is a release, not a configuration change: compare each model with a prompt tuned for it, run the regression, refusal, cost and latency checks, shadow real traffic before canarying, and pin revisions and providers so nobody swaps the model for you."},"body":{"format":"markdown","text":"## What a model swap actually changes\n\nEverything downstream of the weights. A different model has a different chat template, different sampling defaults, different refusal thresholds, different verbosity, a different tokenizer with a different cost per request, different tool-call formatting, and different behaviour on the prompt you tuned for its predecessor.\n\nThe mistake is to treat it as a configuration change. It is a release, and it needs the same discipline as one.\n\n## The frozen prompt problem\n\nYour prompt was tuned against the old model. It contains phrasings, an ordering and an example set that worked there. On a new model, some of it helps and some of it hurts, and there is no way to know which without measuring.\n\nTeams routinely evaluate a new model with the old model's prompt, conclude it is worse, and stay put. Sometimes that is correct. Often the new model with a prompt tuned for it beats both.\n\nSo the honest comparison is **best-of-A against best-of-B**: an hour of prompt tuning on each, on the development split, then compare on the held-out split. Anything less is comparing a tuned system against an untuned one.\n\n## The checklist\n\nBefore any swap reaches users:\n\n1. **Held-out task metric**, both models, identical settings, with the noise floor stated.\n2. **Regression set**: the fifty items from the fine-tuning block, covering general behaviour, other languages, other formats and refusals.\n3. **False-refusal set**, because refusal thresholds vary widely between families and this changes silently.\n4. **Cost per task**, measured, not estimated. The tokenizer changed, so the input token count changed.\n5. **Latency at real concurrency**, p95, TTFT separately.\n6. **Format and tool-call compatibility**, tested against the actual serving stack rather than assumed.\n7. **Licence**, re-checked. The new model may have obligations the old one did not.\n\nSeven checks, most of them scripts you already have from earlier lessons. A morning.\n\n## Rolling it out\n\n**Shadow first.** Send a copy of live traffic to the new model, discard its responses, and store both outputs. After a day you have thousands of paired comparisons on real inputs, at no risk. Sample two hundred and read them. This is the highest-value step in the list and the most often skipped.\n\n**Then canary.** 1% of traffic, then 10%, then 50%. Watch the metrics that would show harm: error rate, retry rate, refusal rate, p95 latency, cost per task, and whatever product signal you have: resolution rate, escalation rate, thumbs-down.\n\n**Define the stop condition in advance.** A specific metric crossing a specific threshold rolls the change back automatically or wakes somebody. Deciding what counts as bad while looking at a graph at 2am is how bad changes stay.\n\n**Keep the old configuration deployable.** Both models, both prompts, both sampler settings, selectable by configuration. Reverting should be a deploy, not a re-tune.\n\n## The swap you did not make\n\nThe earlier lesson on providers described this: an aggregator routing to a cheaper provider, or a provider changing its serving precision or context cap, is a model swap performed on your product without your knowledge.\n\nTwo defences. **Pin the provider and the quantisation** where the platform allows it. And **run a small canary evaluation on a schedule**: twenty items, daily, comparing against recorded expected outputs. It costs pennies and it turns a mysterious quality complaint three weeks later into an alert on the day.\n\nThe same applies to self-hosting with an unpinned revision. `main` moves. A pinned commit hash is the difference between a deployment you can reason about and one that changes when nobody deployed anything.\n\n## Write down why\n\nWhen the swap is done, record the decision: the two numbers, the date, the revisions, the prompt commits, and one sentence about why the new model won. In six months somebody will ask why you are on this model, and the alternative to a written answer is repeating the whole exercise from memory."},"check":{"question":"A team evaluates a new model using their existing prompt, finds it 6 points worse, and abandons the swap. What is the flaw in that decision?","options":[{"text":"Six points is within noise on any evaluation set, so the result was meaningless.","correct":false,"tempting":"Whether six points is noise depends entirely on set size, which is a separate question from the comparison being unfair."},{"text":"They compared a prompt tuned for the old model against an untuned one on the new model, so the gap partly measures prompt fit rather than model quality.","correct":true},{"text":"They should have compared on public benchmarks first, which are less sensitive to prompt differences.","correct":false,"tempting":"Public benchmarks are less relevant to the task, and moving the comparison away from their own data weakens it rather than fixing it."},{"text":"The new model needed fine-tuning before any comparison could be fair.","correct":false,"tempting":"Fine-tuning is a much larger commitment than the hour of prompt tuning that would make the comparison fair."}]},"prev":{"slug":"cost-per-task","title":"Cost per task, not cost per token","url":"https://addaly.com/learn/open-models/cost-per-task"},"next":{"slug":"evaluate-in-an-afternoon","title":"Evaluate a Model on Your Task in an Afternoon","url":"https://addaly.com/learn/open-models/evaluate-in-an-afternoon"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Swapping the model under a live product', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/swapping-the-model (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Swapping the model under a live product","courseTitle":"The Open Model Ecosystem","module":{"index":7,"slug":"judging-models-and-claims","title":"Judging models and claims"},"answer":null,"keyIdea":"A model swap is a release, not a configuration change: compare each model with a prompt tuned for it, run the regression, refusal, cost and latency checks, shadow real traffic before canarying, and pin revisions and providers so nobody swaps the model for you.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-a-model-swap-actually-changes","runs":[{"text":"What a model swap actually changes"}]},{"t":"p","runs":[{"text":"Everything downstream of the weights. A different model has a different chat template, different sampling defaults, different refusal thresholds, different verbosity, a different tokenizer with a different cost per request, different tool-call formatting, and different behaviour on the prompt you tuned for its predecessor."}]},{"t":"p","runs":[{"text":"The mistake is to treat it as a configuration change. It is a release, and it needs the same discipline as one."}]},{"t":"h2","id":"the-frozen-prompt-problem","runs":[{"text":"The frozen prompt problem"}]},{"t":"p","runs":[{"text":"Your prompt was tuned against the old model. It contains phrasings, an ordering and an example set that worked there. On a new model, some of it helps and some of it hurts, and there is no way to know which without measuring."}]},{"t":"p","runs":[{"text":"Teams routinely evaluate a new model with the old model’s prompt, conclude it is worse, and stay put. Sometimes that is correct. Often the new model with a prompt tuned for it beats both."}]},{"t":"p","runs":[{"text":"So the honest comparison is "},{"text":"best-of-A against best-of-B","b":true},{"text":": an hour of prompt tuning on each, on the development split, then compare on the held-out split. Anything less is comparing a tuned system against an untuned one."}]},{"t":"h2","id":"the-checklist","runs":[{"text":"The checklist"}]},{"t":"p","runs":[{"text":"Before any swap reaches users:"}]},{"t":"ol","start":1,"items":[[{"text":"Held-out task metric","b":true},{"text":", both models, identical settings, with the noise floor stated."}],[{"text":"Regression set","b":true},{"text":": the fifty items from the fine-tuning block, covering general behaviour, other languages, other formats and refusals."}],[{"text":"False-refusal set","b":true},{"text":", because refusal thresholds vary widely between families and this changes silently."}],[{"text":"Cost per task","b":true},{"text":", measured, not estimated. The tokenizer changed, so the input token count changed."}],[{"text":"Latency at real concurrency","b":true},{"text":", p95, TTFT separately."}],[{"text":"Format and tool-call compatibility","b":true},{"text":", tested against the actual serving stack rather than assumed."}],[{"text":"Licence","b":true},{"text":", re-checked. The new model may have obligations the old one did not."}]]},{"t":"p","runs":[{"text":"Seven checks, most of them scripts you already have from earlier lessons. A morning."}]},{"t":"h2","id":"rolling-it-out","runs":[{"text":"Rolling it out"}]},{"t":"p","runs":[{"text":"Shadow first.","b":true},{"text":" Send a copy of live traffic to the new model, discard its responses, and store both outputs. After a day you have thousands of paired comparisons on real inputs, at no risk. Sample two hundred and read them. This is the highest-value step in the list and the most often skipped."}]},{"t":"p","runs":[{"text":"Then canary.","b":true},{"text":" 1% of traffic, then 10%, then 50%. Watch the metrics that would show harm: error rate, retry rate, refusal rate, p95 latency, cost per task, and whatever product signal you have: resolution rate, escalation rate, thumbs-down."}]},{"t":"p","runs":[{"text":"Define the stop condition in advance.","b":true},{"text":" A specific metric crossing a specific threshold rolls the change back automatically or wakes somebody. Deciding what counts as bad while looking at a graph at 2am is how bad changes stay."}]},{"t":"p","runs":[{"text":"Keep the old configuration deployable.","b":true},{"text":" Both models, both prompts, both sampler settings, selectable by configuration. Reverting should be a deploy, not a re-tune."}]},{"t":"h2","id":"the-swap-you-did-not-make","runs":[{"text":"The swap you did not make"}]},{"t":"p","runs":[{"text":"The earlier lesson on providers described this: an aggregator routing to a cheaper provider, or a provider changing its serving precision or context cap, is a model swap performed on your product without your knowledge."}]},{"t":"p","runs":[{"text":"Two defences. "},{"text":"Pin the provider and the quantisation","b":true},{"text":" where the platform allows it. And "},{"text":"run a small canary evaluation on a schedule","b":true},{"text":": twenty items, daily, comparing against recorded expected outputs. It costs pennies and it turns a mysterious quality complaint three weeks later into an alert on the day."}]},{"t":"p","runs":[{"text":"The same applies to self-hosting with an unpinned revision. "},{"text":"main","code":true},{"text":" moves. A pinned commit hash is the difference between a deployment you can reason about and one that changes when nobody deployed anything."}]},{"t":"h2","id":"write-down-why","runs":[{"text":"Write down why"}]},{"t":"p","runs":[{"text":"When the swap is done, record the decision: the two numbers, the date, the revisions, the prompt commits, and one sentence about why the new model won. In six months somebody will ask why you are on this model, and the alternative to a written answer is repeating the whole exercise from memory."}]}],"check":{"question":[{"text":"A team evaluates a new model using their existing prompt, finds it 6 points worse, and abandons the swap. What is the flaw in that decision?"}],"options":[{"text":[{"text":"Six points is within noise on any evaluation set, so the result was meaningless."}],"correct":false,"tempting":[{"text":"Whether six points is noise depends entirely on set size, which is a separate question from the comparison being unfair."}]},{"text":[{"text":"They compared a prompt tuned for the old model against an untuned one on the new model, so the gap partly measures prompt fit rather than model quality."}],"correct":true,"tempting":null},{"text":[{"text":"They should have compared on public benchmarks first, which are less sensitive to prompt differences."}],"correct":false,"tempting":[{"text":"Public benchmarks are less relevant to the task, and moving the comparison away from their own data weakens it rather than fixing it."}]},{"text":[{"text":"The new model needed fine-tuning before any comparison could be fair."}],"correct":false,"tempting":[{"text":"Fine-tuning is a much larger commitment than the hour of prompt tuning that would make the comparison fair."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"cost-per-task","title":"Cost per task, not cost per token","minutes":8},"next":{"slug":"evaluate-in-an-afternoon","title":"Evaluate a Model on Your Task in an Afternoon","minutes":10}}}