{"course":{"slug":"agents-and-automation","title":"Agents and Automation","url":"https://addaly.com/learn/agents-and-automation","level":"intermediate","track":"build"},"lesson":{"slug":"pinning-and-migrating-models","title":"The model you tested is not the model you deployed","url":"https://addaly.com/learn/agents-and-automation/pinning-and-migrating-models","position":60,"of":72,"minutes":9,"answer":"Log the model id the provider returns rather than the one you sent, because an alias resolves to a different version over time and that difference is the explanation you will be looking for.","keyIdea":"Log the model id the provider returns rather than the one you sent, because an alias resolves to a different version over time and that difference is the explanation you will be looking for."},"body":{"format":"markdown","text":"## Aliases move\n\nMost providers offer two ways to name a model: a friendly alias, and a pinned identifier with a date or version in it. The alias is convenient and it points somewhere different over time. One morning your prompt is being served by a different model, your outputs change, nothing in your repository changed, and the git log is no help at all.\n\nPin the identifier. Then do the thing that makes the pin useful: **log the model id the API returns in the response**, not the one you sent. They can differ (an alias resolves, a deprecated version redirects) and the returned value is the only record of what actually ran.\n\nPinning is a deferral, not an escape. Pinned versions get deprecated, typically with a few months' notice rather than years. Put the retirement date in a shared calendar with a fortnight's warning the moment you pin, because the alternative is discovering it from a 404 in production.\n\n## Treat a model change as a dependency upgrade\n\nYour prompt is fitted to a particular model. Between versions, any of these can move: how readily it calls tools, how verbose it is, how it formats lists, when it refuses, how it handles a long transcript, and occasionally the tokeniser, which shifts your costs and your truncation points.\n\nSo a model change is a major-version bump of your most important dependency. It gets the same treatment: run the golden task set on both, compare per task rather than in aggregate, read the diffs of tasks that changed, then canary.\n\nThe tasks that regress are usually specific and fixable: an instruction the old model inferred and the new one wants stated. Twenty minutes of reading beats a week of arguing about whether the new model is better.\n\n## Version everything as one thing\n\nThree things determine behaviour: the model id, the prompt version, and the tool schema version. Combine them into a single **config version** and log it on every run.\n\n> Figure: One config version, three moving parts. Layers, top to bottom: Model id, pinned (The dated identifier, and log the id the response returns rather than the one you sent.); Prompt version (A commit hash. A two-minute edit in a vendor's web console is an untracked production deployment.); Tool schema version (Change what a tool returns and you change behaviour, and quietly invalidate every evaluation you have run.). Somebody edited a tool description on Tuesday, somebody changed the model on Wednesday, quality fell on Thursday. With one version stamped on every run the comparison takes a minute; without it, attribution is impossible.\n\nWithout this, attribution is impossible. Somebody edited a tool description on Tuesday, somebody else changed the model on Wednesday, quality fell on Thursday, and there is no way to say which. With it, you filter runs by config version and the comparison takes a minute.\n\nKeep the previous config deployed rather than deleted, so rollback is changing a value rather than finding an old file.\n\n## Reproducibility\n\nTemperature zero and a fixed seed reduce variation. They do not guarantee identical outputs. Providers batch requests together, run on varied hardware, and update inference kernels, and floating-point non-associativity means the same logits are not reproduced bit for bit. Most providers document this as best-effort rather than a guarantee.\n\nSo do not build anything that depends on identical outputs from the same input: a cache keyed on exact response text, a test asserting a string match, a diff-based check that assumes stability. Assert on properties: valid schema, correct field values, the right tool called. Those hold; byte equality does not, and a suite built on it will fail intermittently for reasons nobody can reproduce.\n\n## Fallback chains that have never been tested are not fallbacks\n\nChaining providers so a second is tried when the first fails is sensible, and it is more work than adding an `except` block. Each provider needs its own prompt shaping, its own tool-calling format, its own token limits and its own quirks around structured output. A chain assembled on the assumption that the interfaces are interchangeable produces a fallback that fails the moment it is needed.\n\nTwo habits make it real. Run the golden set against every provider in the chain, not just the primary, and keep the results. And exercise the fallback path deliberately on a schedule (force a failover once a month during working hours) because an untested recovery path degrades silently while the primary keeps working.\n\n## Say what you ran\n\nWhen somebody asks why an output looks different from last month, the answer should be a query: this run used config version 14, that one used version 11, here are the three things that differ. That capability costs one extra column and it is the difference between a technical answer and a shrug, which in a regulated setting is the difference between a finding and a conversation."},"check":{"question":"A team pins a dated model identifier in their code but logs only the string they sent. Six weeks later, outputs have clearly changed and nothing in the repository has. What did the logging omission cost them?","options":[{"text":"The token counts, which would have shown whether prompts had grown.","correct":false,"tempting":"Useful telemetry, but token growth explains cost and truncation rather than a change in the character of the outputs."},{"text":"The ability to reproduce runs exactly, which a logged seed would have restored.","correct":false,"tempting":"Seeds reduce variation but providers do not guarantee bit-identical outputs, so exact reproduction was never available."},{"text":"The record of which model actually served each request, since a pinned id can be redirected on deprecation and the response is the only place that is visible.","correct":true},{"text":"The prompt version, which is the more likely cause of a change in behaviour.","correct":false,"tempting":"Prompt versioning matters just as much, but it is a separate omission: here the question is what the missing model id specifically hid."}]},"prev":{"slug":"queues-and-concurrency","title":"Rate limits, workers and the thundering herd","url":"https://addaly.com/learn/agents-and-automation/queues-and-concurrency"},"next":{"slug":"rollout-and-rollback","title":"Shadow, canary, and the off switch","url":"https://addaly.com/learn/agents-and-automation/rollout-and-rollback"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The model you tested is not the model you deployed', in Agents and Automation. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/agents-and-automation/pinning-and-migrating-models (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The model you tested is not the model you deployed","courseTitle":"Agents and Automation","module":{"index":7,"slug":"shipping-and-operating","title":"Shipping it, and keeping it alive"},"answer":null,"keyIdea":"Log the model id the provider returns rather than the one you sent, because an alias resolves to a different version over time and that difference is the explanation you will be looking for.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"One config version, three moving parts","layers":[{"label":"Model id, pinned","detail":"The dated identifier, and log the id the response returns rather than the one you sent."},{"label":"Prompt version","detail":"A commit hash. A two-minute edit in a vendor’s web console is an untracked production deployment."},{"label":"Tool schema version","detail":"Change what a tool returns and you change behaviour, and quietly invalidate every evaluation you have run."}],"caption":"Somebody edited a tool description on Tuesday, somebody changed the model on Wednesday, quality fell on Thursday. With one version stamped on every run the comparison takes a minute; without it, attribution is impossible."},"alt":"Figure: One config version, three moving parts. Layers, top to bottom: Model id, pinned (The dated identifier, and log the id the response returns rather than the one you sent.); Prompt version (A commit hash. A two-minute edit in a vendor’s web console is an untracked production deployment.); Tool schema version (Change what a tool returns and you change behaviour, and quietly invalidate every evaluation you have run.). Somebody edited a tool description on Tuesday, somebody changed the model on Wednesday, quality fell on Thursday. With one version stamped on every run the comparison takes a minute; without it, attribution is impossible."}},"blocks":[{"t":"h2","id":"aliases-move","runs":[{"text":"Aliases move"}]},{"t":"p","runs":[{"text":"Most providers offer two ways to name a model: a friendly alias, and a pinned identifier with a date or version in it. The alias is convenient and it points somewhere different over time. One morning your prompt is being served by a different model, your outputs change, nothing in your repository changed, and the git log is no help at all."}]},{"t":"p","runs":[{"text":"Pin the identifier. Then do the thing that makes the pin useful: "},{"text":"log the model id the API returns in the response","b":true},{"text":", not the one you sent. They can differ (an alias resolves, a deprecated version redirects) and the returned value is the only record of what actually ran."}]},{"t":"p","runs":[{"text":"Pinning is a deferral, not an escape. Pinned versions get deprecated, typically with a few months’ notice rather than years. Put the retirement date in a shared calendar with a fortnight’s warning the moment you pin, because the alternative is discovering it from a 404 in production."}]},{"t":"h2","id":"treat-a-model-change-as-a-dependency-upgrade","runs":[{"text":"Treat a model change as a dependency upgrade"}]},{"t":"p","runs":[{"text":"Your prompt is fitted to a particular model. Between versions, any of these can move: how readily it calls tools, how verbose it is, how it formats lists, when it refuses, how it handles a long transcript, and occasionally the tokeniser, which shifts your costs and your truncation points."}]},{"t":"p","runs":[{"text":"So a model change is a major-version bump of your most important dependency. It gets the same treatment: run the golden task set on both, compare per task rather than in aggregate, read the diffs of tasks that changed, then canary."}]},{"t":"p","runs":[{"text":"The tasks that regress are usually specific and fixable: an instruction the old model inferred and the new one wants stated. Twenty minutes of reading beats a week of arguing about whether the new model is better."}]},{"t":"h2","id":"version-everything-as-one-thing","runs":[{"text":"Version everything as one thing"}]},{"t":"p","runs":[{"text":"Three things determine behaviour: the model id, the prompt version, and the tool schema version. Combine them into a single "},{"text":"config version","b":true},{"text":" and log it on every run."}]},{"t":"p","runs":[{"text":"Without this, attribution is impossible. Somebody edited a tool description on Tuesday, somebody else changed the model on Wednesday, quality fell on Thursday, and there is no way to say which. With it, you filter runs by config version and the comparison takes a minute."}]},{"t":"p","runs":[{"text":"Keep the previous config deployed rather than deleted, so rollback is changing a value rather than finding an old file."}]},{"t":"h2","id":"reproducibility","runs":[{"text":"Reproducibility"}]},{"t":"p","runs":[{"text":"Temperature zero and a fixed seed reduce variation. They do not guarantee identical outputs. Providers batch requests together, run on varied hardware, and update inference kernels, and floating-point non-associativity means the same logits are not reproduced bit for bit. Most providers document this as best-effort rather than a guarantee."}]},{"t":"p","runs":[{"text":"So do not build anything that depends on identical outputs from the same input: a cache keyed on exact response text, a test asserting a string match, a diff-based check that assumes stability. Assert on properties: valid schema, correct field values, the right tool called. Those hold; byte equality does not, and a suite built on it will fail intermittently for reasons nobody can reproduce."}]},{"t":"h2","id":"fallback-chains-that-have-never-been-tested-are-not-fallback","runs":[{"text":"Fallback chains that have never been tested are not fallbacks"}]},{"t":"p","runs":[{"text":"Chaining providers so a second is tried when the first fails is sensible, and it is more work than adding an "},{"text":"except","code":true},{"text":" block. Each provider needs its own prompt shaping, its own tool-calling format, its own token limits and its own quirks around structured output. A chain assembled on the assumption that the interfaces are interchangeable produces a fallback that fails the moment it is needed."}]},{"t":"p","runs":[{"text":"Two habits make it real. Run the golden set against every provider in the chain, not just the primary, and keep the results. And exercise the fallback path deliberately on a schedule (force a failover once a month during working hours) because an untested recovery path degrades silently while the primary keeps working."}]},{"t":"h2","id":"say-what-you-ran","runs":[{"text":"Say what you ran"}]},{"t":"p","runs":[{"text":"When somebody asks why an output looks different from last month, the answer should be a query: this run used config version 14, that one used version 11, here are the three things that differ. That capability costs one extra column and it is the difference between a technical answer and a shrug, which in a regulated setting is the difference between a finding and a conversation."}]}],"check":{"question":[{"text":"A team pins a dated model identifier in their code but logs only the string they sent. Six weeks later, outputs have clearly changed and nothing in the repository has. What did the logging omission cost them?"}],"options":[{"text":[{"text":"The token counts, which would have shown whether prompts had grown."}],"correct":false,"tempting":[{"text":"Useful telemetry, but token growth explains cost and truncation rather than a change in the character of the outputs."}]},{"text":[{"text":"The ability to reproduce runs exactly, which a logged seed would have restored."}],"correct":false,"tempting":[{"text":"Seeds reduce variation but providers do not guarantee bit-identical outputs, so exact reproduction was never available."}]},{"text":[{"text":"The record of which model actually served each request, since a pinned id can be redirected on deprecation and the response is the only place that is visible."}],"correct":true,"tempting":null},{"text":[{"text":"The prompt version, which is the more likely cause of a change in behaviour."}],"correct":false,"tempting":[{"text":"Prompt versioning matters just as much, but it is a separate omission: here the question is what the missing model id specifically hid."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"queues-and-concurrency","title":"Rate limits, workers and the thundering herd","minutes":9},"next":{"slug":"rollout-and-rollback","title":"Shadow, canary, and the off switch","minutes":9}}}