{"course":{"slug":"ai-actually-explained","title":"AI, Actually Explained","url":"https://addaly.com/learn/ai-actually-explained","level":"beginner","track":"start"},"lesson":{"slug":"the-model-changed-under-you","title":"The thing you tested is not the thing running today","url":"https://addaly.com/learn/ai-actually-explained/the-model-changed-under-you","position":39,"of":92,"minutes":8,"answer":"Consumer AI products change silently and route between models, so \"it got worse\" is unsettleable without your own small set of real test cases run before and after.","keyIdea":"Consumer AI products change silently and route between models, so \"it got worse\" is unsettleable without your own small set of real test cases run before and after."},"body":{"format":"markdown","text":"## No version number on the door\n\nWhen a piece of ordinary software updates, you usually know. There is a version, a changelog, and often a choice about when to take it.\n\nConsumer AI products work differently. The model behind a chat interface is changed without announcement, on the provider's schedule. Sometimes the name stays the same while the model behind it does not. Sometimes several models sit behind one name, and which one you get depends on load, region, your subscription tier, or a routing model deciding how hard your question looks.\n\nOn top of that, providers run experiments. Two people asking the same question at the same moment may be served by different systems, because one of them is in a test group.\n\nSo \"the model\" you evaluated last month is not a stable object, and the sentence \"it used to be better at this\" is sometimes exactly true.\n\n## Is the perceived degradation real?\n\nHonestly: sometimes yes, sometimes no, and it is hard to tell from the inside.\n\n**Real causes.** Providers optimise for cost, and cheaper serving (smaller models, more aggressive quantisation, shorter effective context) can reduce quality on specific tasks while overall benchmark scores hold up or improve. Safety adjustments can make a model refuse or hedge where it previously answered. A change that improves nine tasks can degrade the tenth, and yours may be the tenth.\n\n**Perceptual causes.** Your prompts drift. Your standards rise as you get used to the tool. The variation between runs was always there, and a run of three poor answers is unremarkable in a system with a ten per cent failure rate. Early enthusiasm makes early outputs look better in memory than they were.\n\nWithout your own record you cannot distinguish these, and almost nobody keeps one. Which leads to the practical part of this lesson.\n\n## Keep a small personal test set\n\nThis is the single most useful professional habit in this course, and it takes twenty minutes to set up.\n\nWrite down five to ten tasks that represent what you actually use the tool for. Real inputs, not toy ones. For each, record what a good answer looks like: not word for word, but the two or three things it must contain or must not do.\n\nRun them when something feels different. Run them before you commit to a new tool. Run them after a provider announces a change.\n\nWhat this gives you is the ability to say \"this got worse at extracting dates from scanned invoices\" instead of \"it feels worse\". The first is actionable, reportable and checkable. The second is an argument nobody can settle.\n\nIt also protects against the opposite error: abandoning a tool that is fine because you hit three bad runs in a row.\n\n## If you are building something on top\n\nThree rules, all learned expensively by people who skipped them.\n\n**Pin the version.** APIs offer dated model identifiers rather than a floating alias. Use the dated one, so that changes happen when you choose. The floating alias is convenient right up to the morning your output format changes.\n\n**Watch for deprecation notices.** Pinned models are eventually retired, usually with months of notice sent to the account's registered address. That address is often a personal inbox nobody reads, and the first sign of the problem is a production failure.\n\n**Test before you switch.** Run your test set against the new version alongside the old one. A newer, better model can still be worse at your particular task, especially if your prompts were tuned to the old one's habits.\n\n## And a note on your own prompts\n\nWhen something stops working, the model is the natural suspect and often the wrong one. Check first whether your input changed: a longer document, a different file format, a new field, an extra instruction added weeks ago and forgotten. In practice, changed inputs explain more failures than changed models.\n\nKeeping the prompt in a file with a date and a note about why it says what it says costs nothing and answers this question in seconds."},"check":{"question":"A team reports that a summarisation feature has degraded. What is the most useful first step?","options":[{"text":"Switch to a different provider, since the current one has evidently changed its model.","correct":false,"tempting":"Acts on an unverified diagnosis and discards a working setup on the basis of an impression."},{"text":"Raise the temperature setting, since flatter output suggests the sampler has changed.","correct":false,"tempting":"Adjusts a parameter before establishing what changed, and would alter behaviour without diagnosing anything."},{"text":"Run a fixed set of real inputs whose good answers were recorded earlier, and compare, so that the change is measured rather than felt.","correct":true},{"text":"Ask the model whether it has been updated recently, since it can report its own version.","correct":false,"tempting":"Treats the model as a source of facts about itself, which it has no access to."}]},"prev":{"slug":"text-that-gives-it-orders","title":"When the document tells the model what to do","url":"https://addaly.com/learn/ai-actually-explained/text-that-gives-it-orders"},"next":{"slug":"deepfakes-and-evidence","title":"Fakes, detectors that do not work, and what is left","url":"https://addaly.com/learn/ai-actually-explained/deepfakes-and-evidence"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The thing you tested is not the thing running today', in AI, Actually Explained. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-actually-explained/the-model-changed-under-you (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The thing you tested is not the thing running today","courseTitle":"AI, Actually Explained","module":{"index":4,"slug":"why-it-goes-wrong","title":"Why it gets things wrong"},"answer":null,"keyIdea":"Consumer AI products change silently and route between models, so “it got worse” is unsettleable without your own small set of real test cases run before and after.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"no-version-number-on-the-door","runs":[{"text":"No version number on the door"}]},{"t":"p","runs":[{"text":"When a piece of ordinary software updates, you usually know. There is a version, a changelog, and often a choice about when to take it."}]},{"t":"p","runs":[{"text":"Consumer AI products work differently. The model behind a chat interface is changed without announcement, on the provider’s schedule. Sometimes the name stays the same while the model behind it does not. Sometimes several models sit behind one name, and which one you get depends on load, region, your subscription tier, or a routing model deciding how hard your question looks."}]},{"t":"p","runs":[{"text":"On top of that, providers run experiments. Two people asking the same question at the same moment may be served by different systems, because one of them is in a test group."}]},{"t":"p","runs":[{"text":"So “the model” you evaluated last month is not a stable object, and the sentence “it used to be better at this” is sometimes exactly true."}]},{"t":"h2","id":"is-the-perceived-degradation-real","runs":[{"text":"Is the perceived degradation real?"}]},{"t":"p","runs":[{"text":"Honestly: sometimes yes, sometimes no, and it is hard to tell from the inside."}]},{"t":"p","runs":[{"text":"Real causes.","b":true},{"text":" Providers optimise for cost, and cheaper serving (smaller models, more aggressive quantisation, shorter effective context) can reduce quality on specific tasks while overall benchmark scores hold up or improve. Safety adjustments can make a model refuse or hedge where it previously answered. A change that improves nine tasks can degrade the tenth, and yours may be the tenth."}]},{"t":"p","runs":[{"text":"Perceptual causes.","b":true},{"text":" Your prompts drift. Your standards rise as you get used to the tool. The variation between runs was always there, and a run of three poor answers is unremarkable in a system with a ten per cent failure rate. Early enthusiasm makes early outputs look better in memory than they were."}]},{"t":"p","runs":[{"text":"Without your own record you cannot distinguish these, and almost nobody keeps one. Which leads to the practical part of this lesson."}]},{"t":"h2","id":"keep-a-small-personal-test-set","runs":[{"text":"Keep a small personal test set"}]},{"t":"p","runs":[{"text":"This is the single most useful professional habit in this course, and it takes twenty minutes to set up."}]},{"t":"p","runs":[{"text":"Write down five to ten tasks that represent what you actually use the tool for. Real inputs, not toy ones. For each, record what a good answer looks like: not word for word, but the two or three things it must contain or must not do."}]},{"t":"p","runs":[{"text":"Run them when something feels different. Run them before you commit to a new tool. Run them after a provider announces a change."}]},{"t":"p","runs":[{"text":"What this gives you is the ability to say “this got worse at extracting dates from scanned invoices” instead of “it feels worse”. The first is actionable, reportable and checkable. The second is an argument nobody can settle."}]},{"t":"p","runs":[{"text":"It also protects against the opposite error: abandoning a tool that is fine because you hit three bad runs in a row."}]},{"t":"h2","id":"if-you-are-building-something-on-top","runs":[{"text":"If you are building something on top"}]},{"t":"p","runs":[{"text":"Three rules, all learned expensively by people who skipped them."}]},{"t":"p","runs":[{"text":"Pin the version.","b":true},{"text":" APIs offer dated model identifiers rather than a floating alias. Use the dated one, so that changes happen when you choose. The floating alias is convenient right up to the morning your output format changes."}]},{"t":"p","runs":[{"text":"Watch for deprecation notices.","b":true},{"text":" Pinned models are eventually retired, usually with months of notice sent to the account’s registered address. That address is often a personal inbox nobody reads, and the first sign of the problem is a production failure."}]},{"t":"p","runs":[{"text":"Test before you switch.","b":true},{"text":" Run your test set against the new version alongside the old one. A newer, better model can still be worse at your particular task, especially if your prompts were tuned to the old one’s habits."}]},{"t":"h2","id":"and-a-note-on-your-own-prompts","runs":[{"text":"And a note on your own prompts"}]},{"t":"p","runs":[{"text":"When something stops working, the model is the natural suspect and often the wrong one. Check first whether your input changed: a longer document, a different file format, a new field, an extra instruction added weeks ago and forgotten. In practice, changed inputs explain more failures than changed models."}]},{"t":"p","runs":[{"text":"Keeping the prompt in a file with a date and a note about why it says what it says costs nothing and answers this question in seconds."}]}],"check":{"question":[{"text":"A team reports that a summarisation feature has degraded. What is the most useful first step?"}],"options":[{"text":[{"text":"Switch to a different provider, since the current one has evidently changed its model."}],"correct":false,"tempting":[{"text":"Acts on an unverified diagnosis and discards a working setup on the basis of an impression."}]},{"text":[{"text":"Raise the temperature setting, since flatter output suggests the sampler has changed."}],"correct":false,"tempting":[{"text":"Adjusts a parameter before establishing what changed, and would alter behaviour without diagnosing anything."}]},{"text":[{"text":"Run a fixed set of real inputs whose good answers were recorded earlier, and compare, so that the change is measured rather than felt."}],"correct":true,"tempting":null},{"text":[{"text":"Ask the model whether it has been updated recently, since it can report its own version."}],"correct":false,"tempting":[{"text":"Treats the model as a source of facts about itself, which it has no access to."}]}],"order":[0,1,3,2],"why":null},"prev":{"slug":"text-that-gives-it-orders","title":"When the document tells the model what to do","minutes":9},"next":{"slug":"deepfakes-and-evidence","title":"Fakes, detectors that do not work, and what is left","minutes":9}}}