{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"drift-between-versions","title":"Drift: the same name, a different model","url":"https://addaly.com/learn/how-llms-work/drift-between-versions","position":56,"of":78,"minutes":8,"answer":"A model name is a pointer that providers move, so behaviour changes have to be separated into jitter, version drift and input drift by test rather than by guess.","keyIdea":"A model name is a pointer that providers move, so behaviour changes have to be separated into jitter, version drift and input drift by test rather than by guess, and only a pinned snapshot with a scheduled canary set makes a change attributable."},"body":{"format":"markdown","text":"## The other half of the complaint\n\nModule three explained why a model at temperature zero gives different answers on different days: batching, floating-point arithmetic, near-ties. That is jitter, and it is small.\n\nThere is a second thing people mean by \"the model changed\", and it is not small. The weights behind a name were replaced. The same prompt that produced a clean JSON object in March produces a paragraph of explanation followed by the JSON in June, and every parser downstream breaks. Nothing in your code changed, nothing in your data changed, and the model is still called the same thing.\n\nThe two need different diagnoses, and confusing them wastes weeks.\n\n## The evidence\n\nThe best-known measurement came in 2023, when researchers ran the same prompts against two dated versions of a widely used model three months apart. On one task (deciding whether a number is prime) the reported accuracy swung by tens of points between the versions. On a code-generation task, the fraction of outputs that ran directly fell sharply.\n\nThe study was argued over, and the argument is instructive. Part of the code drop was that the newer version had started wrapping code in markdown fences, which the study's harness did not strip: a formatting change, not a capability change. Part of the prime-number swing was the model becoming more or less willing to show working. Critics were right that some of the change was not what it looked like. They were not able to make it go away. The behaviour under a fixed name had changed enough to break real pipelines, and that is the whole point.\n\nProviders now publish dated snapshots (a model name with a date suffix) alongside floating aliases that point to whatever is current. The alias is the moving target. The snapshot is the thing you can build on, until it is deprecated, which is typically announced months in advance.\n\n## Three kinds of drift, three tests\n\n**Run-to-run jitter.** Same version, same input, output varies. Test: send the same prompt twenty times and look at the spread. Module three's lesson has the causes. This sets your noise band, and you need it to detect the other two.\n\n**Version drift.** The provider changed something. Test: run a fixed set against the dated snapshot and against the alias. A difference beyond your noise band is version drift, attributable to a specific change on a specific date.\n\n**Input drift.** Your users changed. A marketing campaign, a new region, a new use somebody discovered. Test: compare the distribution of inputs this month with three months ago: length, language, intent mix. The evaluation course covers monitoring for this properly; here the point is that it looks like model drift from the output side and is not.\n\n## What actually changes in a version\n\nMore than the weights, and each part leaves a signature.\n\n- **Post-training.** New preference data, new safety data. Signature: tone, hedging, refusal rate, verbosity change together.\n- **The hidden system prompt.** Providers prepend instructions you never see. Signature: formatting habits change (fences, headers, bullet style) with no change in underlying accuracy.\n- **The chat template or tokenizer.** Rare, and it shifts token counts and cache behaviour. Signature: costs change for identical text.\n- **Classifiers around the model.** Separate systems that block or rewrite. Signature: specific inputs start failing with refusals that read differently from the model's own.\n- **Default sampling parameters.** Signature: variance changes with no change in the mean.\n- **Serving-side quantisation.** Signature: small, broad degradation, worst on arithmetic and long-context recall, discussed in module seven.\n\nKnowing the signatures lets you say *what* moved rather than \"it got worse\".\n\n> Figure: One drift incident, as the log records it. Timeline: Day 0, A canary set of 200 items runs nightly. Pass rate 94%, noise band plus or minus 3 points, measured by sending the same prompts twenty times.; Day 41, 03:00, Pass rate 71%. Nothing in the repository changed that week and no data changed either.; Day 41, 09:00, Format checks show 58 of 200 replies now wrap the JSON in a markdown fence. Accuracy on the parsed remainder is unchanged.; Day 41, 10:00, The dated snapshot still scores 94%; the floating alias scores 71%. Version drift, attributable to a specific change on a specific date.; Day 41, 11:00, Pin the snapshot, strip fences in the parser. Both fixes ship the same morning.; Day 132, Deprecation notice for the snapshot, three months ahead. Re-tuning the prompt is scheduled work rather than an outage.. The noise band came first: without it, day 41 is an argument rather than a measurement. The signature (formatting changed, accuracy did not) named the component before anybody opened a support ticket.\n\n## Open weights do not drift; serving stacks do\n\nA downloaded weight file is fixed forever, and that is the strongest argument for running your own model where drift is intolerable. But the stack around it moves. Inference engines change sampling defaults and template handling between releases. A quantised file on a sharing site can be re-uploaded under the same name with a different recipe. The fix is to pin three things: the file's hash, the engine version, and the chat template text. Pin any two and the third will find you.\n\n## The practices, with their reasons\n\n1. **Build against dated snapshots.** The alias is for experiments.\n2. **Keep a canary set.** One to three hundred items with expected outputs, run on a schedule. Alert when the score leaves the noise band you measured. Include format checks (is the JSON valid, is there a fence) because most reported drift is format.\n3. **Log the version with every call.** Model id, snapshot date, template hash, sampling parameters. Without this, a change cannot be attributed and every investigation starts from zero.\n4. **Budget for migration.** A deprecation notice means the prompt tuned to this version (the previous lesson explained how specific such tuning is) will need re-tuning against the next. That is a scheduled cost, not a surprise.\n5. **Read release notes as hypotheses.** They tell you what the provider chose to mention.\n\n## What cannot be done\n\nYou cannot stop a provider changing a model, and there is no changelog fine-grained enough to tell you everything that moved. What you can do is detect the change within a day, attribute it to a component, and have already decided what to do about it. With open weights you can freeze the world, at the cost of running it yourself, which is where the next module begins."},"check":{"question":"A pipeline that parsed a model's code output starts failing on 40 per cent of requests with no change to the code or the inputs. Investigation shows the model now wraps its code in markdown fences. What is the diagnosis, and the right response?","options":[{"text":"Run-to-run jitter, so the fix is to retry each failed request until the model produces unfenced code.","correct":false,"tempting":"Jitter does cause intermittent failures, but a consistent new formatting habit across 40 per cent of traffic is a change in the model, not noise, and retries would just re-sample the same behaviour."},{"text":"Version drift in output format; pin a dated snapshot and put a format check in the canary set.","correct":true},{"text":"Input drift (users have started asking for markdown) so the prompt classifier that routes requests needs retraining on recent traffic.","correct":false,"tempting":"Input drift is a real category and looks similar from the output side, but the question rules it out: the inputs did not change, and the fences appear across the board."},{"text":"The provider's tokenizer changed so that backticks are now a single token, and the fix is to mask that token id during decoding with a constrained grammar.","correct":false,"tempting":"Over-engineers from a tokenisation guess; tokenizer changes are rare and would show up as changed token counts, and a grammar mask treats the symptom while leaving the version unpinned."}]},"prev":{"slug":"prompt-sensitivity","title":"Why a comma changes the answer","url":"https://addaly.com/learn/how-llms-work/prompt-sensitivity"},"next":{"slug":"why-jailbreaks-work","title":"Why jailbreaks work: two failures of safety training","url":"https://addaly.com/learn/how-llms-work/why-jailbreaks-work"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Drift: the same name, a different model', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/drift-between-versions (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Drift: the same name, a different model","courseTitle":"How a Language Model Actually Works","module":{"index":6,"slug":"why-it-goes-wrong","title":"Why it goes wrong, by mechanism"},"answer":null,"keyIdea":"A model name is a pointer that providers move, so behaviour changes have to be separated into jitter, version drift and input drift by test rather than by guess, and only a pinned snapshot with a scheduled canary set makes a change attributable.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"timeline","title":"One drift incident, as the log records it","events":[{"when":"Day 0","what":"A canary set of 200 items runs nightly. Pass rate 94%, noise band plus or minus 3 points, measured by sending the same prompts twenty times."},{"when":"Day 41, 03:00","what":"Pass rate 71%. Nothing in the repository changed that week and no data changed either."},{"when":"Day 41, 09:00","what":"Format checks show 58 of 200 replies now wrap the JSON in a markdown fence. Accuracy on the parsed remainder is unchanged."},{"when":"Day 41, 10:00","what":"The dated snapshot still scores 94%; the floating alias scores 71%. Version drift, attributable to a specific change on a specific date."},{"when":"Day 41, 11:00","what":"Pin the snapshot, strip fences in the parser. Both fixes ship the same morning."},{"when":"Day 132","what":"Deprecation notice for the snapshot, three months ahead. Re-tuning the prompt is scheduled work rather than an outage."}],"caption":"The noise band came first: without it, day 41 is an argument rather than a measurement. The signature (formatting changed, accuracy did not) named the component before anybody opened a support ticket."},"alt":"Figure: One drift incident, as the log records it. Timeline: Day 0, A canary set of 200 items runs nightly. Pass rate 94%, noise band plus or minus 3 points, measured by sending the same prompts twenty times.; Day 41, 03:00, Pass rate 71%. Nothing in the repository changed that week and no data changed either.; Day 41, 09:00, Format checks show 58 of 200 replies now wrap the JSON in a markdown fence. Accuracy on the parsed remainder is unchanged.; Day 41, 10:00, The dated snapshot still scores 94%; the floating alias scores 71%. Version drift, attributable to a specific change on a specific date.; Day 41, 11:00, Pin the snapshot, strip fences in the parser. Both fixes ship the same morning.; Day 132, Deprecation notice for the snapshot, three months ahead. Re-tuning the prompt is scheduled work rather than an outage.. The noise band came first: without it, day 41 is an argument rather than a measurement. The signature (formatting changed, accuracy did not) named the component before anybody opened a support ticket."}},"blocks":[{"t":"h2","id":"the-other-half-of-the-complaint","runs":[{"text":"The other half of the complaint"}]},{"t":"p","runs":[{"text":"Module three explained why a model at temperature zero gives different answers on different days: batching, floating-point arithmetic, near-ties. That is jitter, and it is small."}]},{"t":"p","runs":[{"text":"There is a second thing people mean by “the model changed”, and it is not small. The weights behind a name were replaced. The same prompt that produced a clean JSON object in March produces a paragraph of explanation followed by the JSON in June, and every parser downstream breaks. Nothing in your code changed, nothing in your data changed, and the model is still called the same thing."}]},{"t":"p","runs":[{"text":"The two need different diagnoses, and confusing them wastes weeks."}]},{"t":"h2","id":"the-evidence","runs":[{"text":"The evidence"}]},{"t":"p","runs":[{"text":"The best-known measurement came in 2023, when researchers ran the same prompts against two dated versions of a widely used model three months apart. On one task (deciding whether a number is prime) the reported accuracy swung by tens of points between the versions. On a code-generation task, the fraction of outputs that ran directly fell sharply."}]},{"t":"p","runs":[{"text":"The study was argued over, and the argument is instructive. Part of the code drop was that the newer version had started wrapping code in markdown fences, which the study’s harness did not strip: a formatting change, not a capability change. Part of the prime-number swing was the model becoming more or less willing to show working. Critics were right that some of the change was not what it looked like. They were not able to make it go away. The behaviour under a fixed name had changed enough to break real pipelines, and that is the whole point."}]},{"t":"p","runs":[{"text":"Providers now publish dated snapshots (a model name with a date suffix) alongside floating aliases that point to whatever is current. The alias is the moving target. The snapshot is the thing you can build on, until it is deprecated, which is typically announced months in advance."}]},{"t":"h2","id":"three-kinds-of-drift-three-tests","runs":[{"text":"Three kinds of drift, three tests"}]},{"t":"p","runs":[{"text":"Run-to-run jitter.","b":true},{"text":" Same version, same input, output varies. Test: send the same prompt twenty times and look at the spread. Module three’s lesson has the causes. This sets your noise band, and you need it to detect the other two."}]},{"t":"p","runs":[{"text":"Version drift.","b":true},{"text":" The provider changed something. Test: run a fixed set against the dated snapshot and against the alias. A difference beyond your noise band is version drift, attributable to a specific change on a specific date."}]},{"t":"p","runs":[{"text":"Input drift.","b":true},{"text":" Your users changed. A marketing campaign, a new region, a new use somebody discovered. Test: compare the distribution of inputs this month with three months ago: length, language, intent mix. The evaluation course covers monitoring for this properly; here the point is that it looks like model drift from the output side and is not."}]},{"t":"h2","id":"what-actually-changes-in-a-version","runs":[{"text":"What actually changes in a version"}]},{"t":"p","runs":[{"text":"More than the weights, and each part leaves a signature."}]},{"t":"ul","items":[[{"text":"Post-training.","b":true},{"text":" New preference data, new safety data. Signature: tone, hedging, refusal rate, verbosity change together."}],[{"text":"The hidden system prompt.","b":true},{"text":" Providers prepend instructions you never see. Signature: formatting habits change (fences, headers, bullet style) with no change in underlying accuracy."}],[{"text":"The chat template or tokenizer.","b":true},{"text":" Rare, and it shifts token counts and cache behaviour. Signature: costs change for identical text."}],[{"text":"Classifiers around the model.","b":true},{"text":" Separate systems that block or rewrite. Signature: specific inputs start failing with refusals that read differently from the model’s own."}],[{"text":"Default sampling parameters.","b":true},{"text":" Signature: variance changes with no change in the mean."}],[{"text":"Serving-side quantisation.","b":true},{"text":" Signature: small, broad degradation, worst on arithmetic and long-context recall, discussed in module seven."}]]},{"t":"p","runs":[{"text":"Knowing the signatures lets you say "},{"text":"what","i":true},{"text":" moved rather than “it got worse”."}]},{"t":"h2","id":"open-weights-do-not-drift-serving-stacks-do","runs":[{"text":"Open weights do not drift; serving stacks do"}]},{"t":"p","runs":[{"text":"A downloaded weight file is fixed forever, and that is the strongest argument for running your own model where drift is intolerable. But the stack around it moves. Inference engines change sampling defaults and template handling between releases. A quantised file on a sharing site can be re-uploaded under the same name with a different recipe. The fix is to pin three things: the file’s hash, the engine version, and the chat template text. Pin any two and the third will find you."}]},{"t":"h2","id":"the-practices-with-their-reasons","runs":[{"text":"The practices, with their reasons"}]},{"t":"ol","start":1,"items":[[{"text":"Build against dated snapshots.","b":true},{"text":" The alias is for experiments."}],[{"text":"Keep a canary set.","b":true},{"text":" One to three hundred items with expected outputs, run on a schedule. Alert when the score leaves the noise band you measured. Include format checks (is the JSON valid, is there a fence) because most reported drift is format."}],[{"text":"Log the version with every call.","b":true},{"text":" Model id, snapshot date, template hash, sampling parameters. Without this, a change cannot be attributed and every investigation starts from zero."}],[{"text":"Budget for migration.","b":true},{"text":" A deprecation notice means the prompt tuned to this version (the previous lesson explained how specific such tuning is) will need re-tuning against the next. That is a scheduled cost, not a surprise."}],[{"text":"Read release notes as hypotheses.","b":true},{"text":" They tell you what the provider chose to mention."}]]},{"t":"h2","id":"what-cannot-be-done","runs":[{"text":"What cannot be done"}]},{"t":"p","runs":[{"text":"You cannot stop a provider changing a model, and there is no changelog fine-grained enough to tell you everything that moved. What you can do is detect the change within a day, attribute it to a component, and have already decided what to do about it. With open weights you can freeze the world, at the cost of running it yourself, which is where the next module begins."}]}],"check":{"question":[{"text":"A pipeline that parsed a model’s code output starts failing on 40 per cent of requests with no change to the code or the inputs. Investigation shows the model now wraps its code in markdown fences. What is the diagnosis, and the right response?"}],"options":[{"text":[{"text":"Run-to-run jitter, so the fix is to retry each failed request until the model produces unfenced code."}],"correct":false,"tempting":[{"text":"Jitter does cause intermittent failures, but a consistent new formatting habit across 40 per cent of traffic is a change in the model, not noise, and retries would just re-sample the same behaviour."}]},{"text":[{"text":"Version drift in output format; pin a dated snapshot and put a format check in the canary set."}],"correct":true,"tempting":null},{"text":[{"text":"Input drift (users have started asking for markdown) so the prompt classifier that routes requests needs retraining on recent traffic."}],"correct":false,"tempting":[{"text":"Input drift is a real category and looks similar from the output side, but the question rules it out: the inputs did not change, and the fences appear across the board."}]},{"text":[{"text":"The provider’s tokenizer changed so that backticks are now a single token, and the fix is to mask that token id during decoding with a constrained grammar."}],"correct":false,"tempting":[{"text":"Over-engineers from a tokenisation guess; tokenizer changes are rare and would show up as changed token counts, and a grammar mask treats the symptom while leaving the version unpinned."}]}],"order":[3,0,2,1],"why":null},"prev":{"slug":"prompt-sensitivity","title":"Why a comma changes the answer","minutes":8},"next":{"slug":"why-jailbreaks-work","title":"Why jailbreaks work: two failures of safety training","minutes":9}}}