{"course":{"slug":"choosing-your-tools","title":"Choosing and Using the Tools","url":"https://addaly.com/learn/choosing-your-tools","level":"beginner","track":"work"},"lesson":{"slug":"under-new-management","title":"When the engine gets swapped","url":"https://addaly.com/learn/choosing-your-tools/under-new-management","position":9,"of":76,"minutes":8,"answer":"Tools change underneath you through moving aliases, retuned routers and rewritten system prompts, so keep ten fixed prompts with known answers and a dated note of the model string.","keyIdea":"Tools change underneath you through moving aliases, retuned routers and rewritten system prompts, so keep ten fixed prompts with known answers and a dated note of the model string, and pin versions wherever the work runs unattended."},"body":{"format":"markdown","text":"## The Tuesday problem\n\nA support team has a prompt that has worked for four months. It sorts messages into eight buckets and it is right about 94% of the time. Nobody has touched it. On a Tuesday it starts putting refund requests into the billing bucket, and accuracy falls to 71%.\n\nNothing was deployed. The provider updated the model that the alias in their code points at.\n\nThis is not misconduct and it is not rare. Providers improve models continuously and route the general name to the current build. The new model is better on average and different in the specifics, and your prompt was tuned to the specifics.\n\n## The four ways your tool changes under you\n\n**The alias moves.** `model-4-latest` now points at a new snapshot. Behaviour shifts, sometimes noticeably, on prompts that were near a boundary.\n\n**The router changes.** Consumer products increasingly choose a model per message: cheap one for easy questions, expensive one for hard ones. You cannot see the decision and it is retuned regularly. This is the most common reason a chat app \"feels worse this week\" with no announcement.\n\n**The system prompt changes.** No model change at all, and the assistant becomes more cautious, more verbose, or starts adding a summary you did not ask for.\n\n**A model is deprecated.** The one you liked is withdrawn on a published date, with a suggested successor that is better on benchmarks and different on your work.\n\n## Noticing before your users do\n\nThe whole defence fits on one page and costs nothing.\n\n**Keep ten fixed prompts with known good answers.** Real ones from your work, saved in a plain text file with the answer you consider correct. This is the same file the course keeps returning to. Run them when something feels off. Ten minutes.\n\n**Date every judgement.** When you write down that a tool is good at something, write the date and the model string next to it. A note saying \"handles Marathi well\" is worthless in a year without a date attached.\n\n**Pin the version where it matters.** If you are calling an API for anything that runs unattended, use the dated snapshot rather than the alias, and put the deprecation date in the calendar the day you pin it. You trade a quiet drift for a loud, scheduled migration you control.\n\n**Read the changelog.** Every major provider publishes one. Twenty minutes a month, and it turns most surprises into expected events.\n\n**Subscribe to deprecation notices.** They go to the account email of whoever created the API key, which in a small team is often somebody who has left. Fix that once.\n\n## What to do when it has already happened\n\n1. **Confirm it is real.** Run your ten prompts. If the answers still look right, the change may be in you rather than in the tool: expectations drift too, and so does the difficulty of what you are asking.\n2. **Find out what moved.** Check the changelog and the model string. If the string changed, you have your answer.\n3. **Fix the prompt before switching vendor.** A migration is a week; a prompt adjustment is an hour, and most regressions after a model change are fixed by making implicit expectations explicit. The old model inferred something the new one does not.\n4. **Only then consider moving.** If the successor is worse for your task, this is what your comparison method is for, and it is why keeping a second tool ready is worth the trouble.\n\n## The one that is nobody's fault\n\nSometimes the tool did not change and neither did you: the work did. A prompt written for two-paragraph emails is fed a nine-paragraph one, a classifier trained on last year's ticket vocabulary meets this year's product names, a summariser tuned on tidy PDFs is handed scans. Check the inputs alongside the tool. Roughly a third of the regressions people report to providers turn out to be a change in what they were sending, and the fixed-prompt file is what makes that distinguishable in five minutes rather than a week.\n\n## The limitation\n\nIf you are on a consumer chat plan, you cannot pin anything. There is no version control, no changelog granular enough, and no undo. That is the deal, and it is a real cost of the convenience, not a reason to avoid consumer plans, but a reason to keep your ten prompts and to be sceptical of your own memory. People report tools getting worse constantly, and roughly half the time the file says otherwise.\n\nThat is what the file is for: it is the only witness you have that is not your own recollection of how good something used to be."},"check":{"question":"A team's classification accuracy drops overnight with no deployment on their side. Before switching providers, what is the highest-value step?","options":[{"text":"Re-run the ten fixed prompts, then make the prompt's implicit expectations explicit before switching anything.","correct":true},{"text":"Raise the thinking budget so the new model deliberates longer over each classification.","correct":false,"tempting":"Applies more compute to a task whose failure is a changed interpretation of the prompt, and classification into known buckets rarely benefits from deliberation."},{"text":"Switch every call to a dated snapshot of the previous model so that the old behaviour is restored and locked in permanently.","correct":false,"tempting":"Buys time by freezing behaviour, but pinned snapshots are themselves deprecated on a published date, so it postpones the migration rather than resolving it."},{"text":"Report the regression to the provider and wait for a fix, since the change came from their side.","correct":false,"tempting":"Treats an average improvement as a defect; the provider shipped a better model overall and will not roll it back for one prompt's assumptions."}]},"prev":{"slug":"multimodal-in-practice","title":"'It understands images': what that buys","url":"https://addaly.com/learn/choosing-your-tools/multimodal-in-practice"},"next":{"slug":"real-differences","title":"Where they differ, and where it is marketing","url":"https://addaly.com/learn/choosing-your-tools/real-differences"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'When the engine gets swapped', in Choosing and Using the Tools. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/choosing-your-tools/under-new-management (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"When the engine gets swapped","courseTitle":"Choosing and Using the Tools","module":{"index":1,"slug":"anatomy-of-a-tool","title":"What a tool is made of"},"answer":null,"keyIdea":"Tools change underneath you through moving aliases, retuned routers and rewritten system prompts, so keep ten fixed prompts with known answers and a dated note of the model string, and pin versions wherever the work runs unattended.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-tuesday-problem","runs":[{"text":"The Tuesday problem"}]},{"t":"p","runs":[{"text":"A support team has a prompt that has worked for four months. It sorts messages into eight buckets and it is right about 94% of the time. Nobody has touched it. On a Tuesday it starts putting refund requests into the billing bucket, and accuracy falls to 71%."}]},{"t":"p","runs":[{"text":"Nothing was deployed. The provider updated the model that the alias in their code points at."}]},{"t":"p","runs":[{"text":"This is not misconduct and it is not rare. Providers improve models continuously and route the general name to the current build. The new model is better on average and different in the specifics, and your prompt was tuned to the specifics."}]},{"t":"h2","id":"the-four-ways-your-tool-changes-under-you","runs":[{"text":"The four ways your tool changes under you"}]},{"t":"p","runs":[{"text":"The alias moves.","b":true},{"text":" "},{"text":"model-4-latest","code":true},{"text":" now points at a new snapshot. Behaviour shifts, sometimes noticeably, on prompts that were near a boundary."}]},{"t":"p","runs":[{"text":"The router changes.","b":true},{"text":" Consumer products increasingly choose a model per message: cheap one for easy questions, expensive one for hard ones. You cannot see the decision and it is retuned regularly. This is the most common reason a chat app “feels worse this week” with no announcement."}]},{"t":"p","runs":[{"text":"The system prompt changes.","b":true},{"text":" No model change at all, and the assistant becomes more cautious, more verbose, or starts adding a summary you did not ask for."}]},{"t":"p","runs":[{"text":"A model is deprecated.","b":true},{"text":" The one you liked is withdrawn on a published date, with a suggested successor that is better on benchmarks and different on your work."}]},{"t":"h2","id":"noticing-before-your-users-do","runs":[{"text":"Noticing before your users do"}]},{"t":"p","runs":[{"text":"The whole defence fits on one page and costs nothing."}]},{"t":"p","runs":[{"text":"Keep ten fixed prompts with known good answers.","b":true},{"text":" Real ones from your work, saved in a plain text file with the answer you consider correct. This is the same file the course keeps returning to. Run them when something feels off. Ten minutes."}]},{"t":"p","runs":[{"text":"Date every judgement.","b":true},{"text":" When you write down that a tool is good at something, write the date and the model string next to it. A note saying “handles Marathi well” is worthless in a year without a date attached."}]},{"t":"p","runs":[{"text":"Pin the version where it matters.","b":true},{"text":" If you are calling an API for anything that runs unattended, use the dated snapshot rather than the alias, and put the deprecation date in the calendar the day you pin it. You trade a quiet drift for a loud, scheduled migration you control."}]},{"t":"p","runs":[{"text":"Read the changelog.","b":true},{"text":" Every major provider publishes one. Twenty minutes a month, and it turns most surprises into expected events."}]},{"t":"p","runs":[{"text":"Subscribe to deprecation notices.","b":true},{"text":" They go to the account email of whoever created the API key, which in a small team is often somebody who has left. Fix that once."}]},{"t":"h2","id":"what-to-do-when-it-has-already-happened","runs":[{"text":"What to do when it has already happened"}]},{"t":"ol","start":1,"items":[[{"text":"Confirm it is real.","b":true},{"text":" Run your ten prompts. If the answers still look right, the change may be in you rather than in the tool: expectations drift too, and so does the difficulty of what you are asking."}],[{"text":"Find out what moved.","b":true},{"text":" Check the changelog and the model string. If the string changed, you have your answer."}],[{"text":"Fix the prompt before switching vendor.","b":true},{"text":" A migration is a week; a prompt adjustment is an hour, and most regressions after a model change are fixed by making implicit expectations explicit. The old model inferred something the new one does not."}],[{"text":"Only then consider moving.","b":true},{"text":" If the successor is worse for your task, this is what your comparison method is for, and it is why keeping a second tool ready is worth the trouble."}]]},{"t":"h2","id":"the-one-that-is-nobody-s-fault","runs":[{"text":"The one that is nobody’s fault"}]},{"t":"p","runs":[{"text":"Sometimes the tool did not change and neither did you: the work did. A prompt written for two-paragraph emails is fed a nine-paragraph one, a classifier trained on last year’s ticket vocabulary meets this year’s product names, a summariser tuned on tidy PDFs is handed scans. Check the inputs alongside the tool. Roughly a third of the regressions people report to providers turn out to be a change in what they were sending, and the fixed-prompt file is what makes that distinguishable in five minutes rather than a week."}]},{"t":"h2","id":"the-limitation","runs":[{"text":"The limitation"}],"role":"limit"},{"t":"p","runs":[{"text":"If you are on a consumer chat plan, you cannot pin anything. There is no version control, no changelog granular enough, and no undo. That is the deal, and it is a real cost of the convenience, not a reason to avoid consumer plans, but a reason to keep your ten prompts and to be sceptical of your own memory. People report tools getting worse constantly, and roughly half the time the file says otherwise."}]},{"t":"p","runs":[{"text":"That is what the file is for: it is the only witness you have that is not your own recollection of how good something used to be."}]}],"check":{"question":[{"text":"A team’s classification accuracy drops overnight with no deployment on their side. Before switching providers, what is the highest-value step?"}],"options":[{"text":[{"text":"Re-run the ten fixed prompts, then make the prompt’s implicit expectations explicit before switching anything."}],"correct":true,"tempting":null},{"text":[{"text":"Raise the thinking budget so the new model deliberates longer over each classification."}],"correct":false,"tempting":[{"text":"Applies more compute to a task whose failure is a changed interpretation of the prompt, and classification into known buckets rarely benefits from deliberation."}]},{"text":[{"text":"Switch every call to a dated snapshot of the previous model so that the old behaviour is restored and locked in permanently."}],"correct":false,"tempting":[{"text":"Buys time by freezing behaviour, but pinned snapshots are themselves deprecated on a published date, so it postpones the migration rather than resolving it."}]},{"text":[{"text":"Report the regression to the provider and wait for a fix, since the change came from their side."}],"correct":false,"tempting":[{"text":"Treats an average improvement as a defect; the provider shipped a better model overall and will not roll it back for one prompt’s assumptions."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"multimodal-in-practice","title":"‘It understands images’: what that buys","minutes":8},"next":{"slug":"real-differences","title":"Where they differ, and where it is marketing","minutes":8}}}