The thing you tested is not the thing running today

No version number on the door

When a piece of ordinary software updates, you usually know. There is a version, a changelog, and often a choice about when to take it.

Consumer AI products work differently. The model behind a chat interface is changed without announcement, on the provider’s schedule. Sometimes the name stays the same while the model behind it does not. Sometimes several models sit behind one name, and which one you get depends on load, region, your subscription tier, or a routing model deciding how hard your question looks.

On top of that, providers run experiments. Two people asking the same question at the same moment may be served by different systems, because one of them is in a test group.

So “the model” you evaluated last month is not a stable object, and the sentence “it used to be better at this” is sometimes exactly true.

Is the perceived degradation real?

Honestly: sometimes yes, sometimes no, and it is hard to tell from the inside.

Real causes. Providers optimise for cost, and cheaper serving (smaller models, more aggressive quantisation, shorter effective context) can reduce quality on specific tasks while overall benchmark scores hold up or improve. Safety adjustments can make a model refuse or hedge where it previously answered. A change that improves nine tasks can degrade the tenth, and yours may be the tenth.

Perceptual causes. Your prompts drift. Your standards rise as you get used to the tool. The variation between runs was always there, and a run of three poor answers is unremarkable in a system with a ten per cent failure rate. Early enthusiasm makes early outputs look better in memory than they were.

Without your own record you cannot distinguish these, and almost nobody keeps one. Which leads to the practical part of this lesson.

Keep a small personal test set

This is the single most useful professional habit in this course, and it takes twenty minutes to set up.

Write down five to ten tasks that represent what you actually use the tool for. Real inputs, not toy ones. For each, record what a good answer looks like: not word for word, but the two or three things it must contain or must not do.

Run them when something feels different. Run them before you commit to a new tool. Run them after a provider announces a change.

What this gives you is the ability to say “this got worse at extracting dates from scanned invoices” instead of “it feels worse”. The first is actionable, reportable and checkable. The second is an argument nobody can settle.

It also protects against the opposite error: abandoning a tool that is fine because you hit three bad runs in a row.

If you are building something on top

Three rules, all learned expensively by people who skipped them.

Pin the version. APIs offer dated model identifiers rather than a floating alias. Use the dated one, so that changes happen when you choose. The floating alias is convenient right up to the morning your output format changes.

Watch for deprecation notices. Pinned models are eventually retired, usually with months of notice sent to the account’s registered address. That address is often a personal inbox nobody reads, and the first sign of the problem is a production failure.

Test before you switch. Run your test set against the new version alongside the old one. A newer, better model can still be worse at your particular task, especially if your prompts were tuned to the old one’s habits.

And a note on your own prompts

When something stops working, the model is the natural suspect and often the wrong one. Check first whether your input changed: a longer document, a different file format, a new field, an extra instruction added weeks ago and forgotten. In practice, changed inputs explain more failures than changed models.

Keeping the prompt in a file with a date and a note about why it says what it says costs nothing and answers this question in seconds.