Drift: the same name, a different model

The other half of the complaint

Module three explained why a model at temperature zero gives different answers on different days: batching, floating-point arithmetic, near-ties. That is jitter, and it is small.

There is a second thing people mean by “the model changed”, and it is not small. The weights behind a name were replaced. The same prompt that produced a clean JSON object in March produces a paragraph of explanation followed by the JSON in June, and every parser downstream breaks. Nothing in your code changed, nothing in your data changed, and the model is still called the same thing.

The two need different diagnoses, and confusing them wastes weeks.

The evidence

The best-known measurement came in 2023, when researchers ran the same prompts against two dated versions of a widely used model three months apart. On one task (deciding whether a number is prime) the reported accuracy swung by tens of points between the versions. On a code-generation task, the fraction of outputs that ran directly fell sharply.

The study was argued over, and the argument is instructive. Part of the code drop was that the newer version had started wrapping code in markdown fences, which the study’s harness did not strip: a formatting change, not a capability change. Part of the prime-number swing was the model becoming more or less willing to show working. Critics were right that some of the change was not what it looked like. They were not able to make it go away. The behaviour under a fixed name had changed enough to break real pipelines, and that is the whole point.

Providers now publish dated snapshots (a model name with a date suffix) alongside floating aliases that point to whatever is current. The alias is the moving target. The snapshot is the thing you can build on, until it is deprecated, which is typically announced months in advance.

Three kinds of drift, three tests

Run-to-run jitter. Same version, same input, output varies. Test: send the same prompt twenty times and look at the spread. Module three’s lesson has the causes. This sets your noise band, and you need it to detect the other two.

Version drift. The provider changed something. Test: run a fixed set against the dated snapshot and against the alias. A difference beyond your noise band is version drift, attributable to a specific change on a specific date.

Input drift. Your users changed. A marketing campaign, a new region, a new use somebody discovered. Test: compare the distribution of inputs this month with three months ago: length, language, intent mix. The evaluation course covers monitoring for this properly; here the point is that it looks like model drift from the output side and is not.

What actually changes in a version

More than the weights, and each part leaves a signature.

• Post-training. New preference data, new safety data. Signature: tone, hedging, refusal rate, verbosity change together.
• The hidden system prompt. Providers prepend instructions you never see. Signature: formatting habits change (fences, headers, bullet style) with no change in underlying accuracy.
• The chat template or tokenizer. Rare, and it shifts token counts and cache behaviour. Signature: costs change for identical text.
• Classifiers around the model. Separate systems that block or rewrite. Signature: specific inputs start failing with refusals that read differently from the model’s own.
• Default sampling parameters. Signature: variance changes with no change in the mean.
• Serving-side quantisation. Signature: small, broad degradation, worst on arithmetic and long-context recall, discussed in module seven.

Knowing the signatures lets you say what moved rather than “it got worse”.

Open weights do not drift; serving stacks do

A downloaded weight file is fixed forever, and that is the strongest argument for running your own model where drift is intolerable. But the stack around it moves. Inference engines change sampling defaults and template handling between releases. A quantised file on a sharing site can be re-uploaded under the same name with a different recipe. The fix is to pin three things: the file’s hash, the engine version, and the chat template text. Pin any two and the third will find you.

The practices, with their reasons

1. Build against dated snapshots. The alias is for experiments.
2. Keep a canary set. One to three hundred items with expected outputs, run on a schedule. Alert when the score leaves the noise band you measured. Include format checks (is the JSON valid, is there a fence) because most reported drift is format.
3. Log the version with every call. Model id, snapshot date, template hash, sampling parameters. Without this, a change cannot be attributed and every investigation starts from zero.
4. Budget for migration. A deprecation notice means the prompt tuned to this version (the previous lesson explained how specific such tuning is) will need re-tuning against the next. That is a scheduled cost, not a surprise.
5. Read release notes as hypotheses. They tell you what the provider chose to mention.

What cannot be done

You cannot stop a provider changing a model, and there is no changelog fine-grained enough to tell you everything that moved. What you can do is detect the change within a day, attribute it to a component, and have already decided what to do about it. With open weights you can freeze the world, at the cost of running it yourself, which is where the next module begins.