{"course":{"slug":"cloud-and-shipping","title":"Shipping It","url":"https://addaly.com/learn/cloud-and-shipping","level":"intermediate","track":"build"},"lesson":{"slug":"which-model-answered","title":"Which model, which prompt, which version","url":"https://addaly.com/learn/cloud-and-shipping/which-model-answered","position":52,"of":74,"minutes":9,"answer":"A model system has four independently moving versions (model snapshot, prompt, decoding parameters and retrieval index) so pin each one and record all four with every response.","keyIdea":"A model system has four independently moving versions (model snapshot, prompt, decoding parameters and retrieval index) so pin each one and record all four with every response, or no complaint about last Tuesday can be investigated."},"body":{"format":"markdown","text":"## The complaint you cannot investigate\n\nA user writes: last Tuesday your assistant told me something wrong, here is a screenshot. You open the code. The prompt has been edited twice since. The provider has silently moved their model pointer to a newer snapshot. Somebody changed the temperature. The retrieval index was rebuilt on Wednesday.\n\nYou cannot reproduce the answer, so you cannot say whether it was fixed, and you cannot tell whether the change you are about to make would have prevented it. The deploy lesson gave you a commit SHA for code. A model system needs the same discipline over four more things that all change independently.\n\n## The four versions to record with every response\n\n1. **Model identity, pinned.** Not `gpt-latest` or `claude-latest` or whatever the alias is called this year, but the dated snapshot. Aliases move under you, and when they move, quality changes without a deploy. Pin the version, and treat a provider's deprecation notice as a scheduled piece of work rather than a surprise.\n2. **Prompt version.** The prompt is code. It belongs in the repository, in a file, with a hash, not in a database row somebody edits in an admin panel, and certainly not pasted in three places.\n3. **Decoding parameters.** Temperature, top-p, max tokens, stop sequences, tools available. A temperature of 0.9 makes reproduction impossible in principle; recording the seed where a provider supports one makes it possible again.\n4. **Retrieval state.** If context is fetched, record the index version and the identifiers of the documents that were actually retrieved. \"The model hallucinated\" and \"the retriever returned the wrong document\" look identical from outside and require completely different fixes.\n\nStore all of it on the row with the response, or in a log line with the request ID:\n\n```json\n{\"request_id\":\"c9f2a1\",\"model\":\"provider/model-2026-04-12\",\"prompt\":\"answer_v7:9f31\",\n \"temp\":0.2,\"max_tokens\":600,\"index\":\"kb-2026-08-30\",\"docs\":[\"kb/412\",\"kb/77\"],\n \"in_tokens\":1204,\"out_tokens\":388,\"latency_ms\":2140}\n```\n\nThat single line answers the complaint, feeds the cost breakdown from the previous lesson, and tells you which release a regression started in.\n\n## A registry, for models you train yourself\n\nIf you produce your own models, the artifact is a file, and a file with no provenance is a liability. What to record, at minimum: the training data snapshot or its hash, the code commit, the hyperparameters, the evaluation numbers on the held-out set, who trained it and when.\n\nMLflow is the free, self-hostable standard for this, and its model registry gives you a version number, a stage, and the run that produced it. DVC handles the data-and-artifact half in git. If both feel heavy, a directory named by date and a `model.json` beside every artifact carrying those six fields is a legitimate starting point: the point is that the fields exist, not that a platform holds them.\n\nOne rule: the artifact you evaluated must be the artifact you deploy. Re-training \"the same model\" for production, with the same script and different randomness, produces a different model with unverified numbers. This is the build-once-run-anywhere argument, transferred.\n\n## Evaluate before you ship, on a set that never moves\n\nThis course is not the evaluation course, and the details of building a held-out set belong there. The shipping-side rule is short: a model or prompt change is a deploy, so it needs a check that runs automatically and can block it.\n\nIn practice that is a file of 50 to 300 examples with expected properties, run in CI, reporting a number. Not a perfect measure (nobody has one) but a tripwire. Teams without it discover regressions from users; teams with it discover them in the pull request. The gap between those two is usually one afternoon of work.\n\nRecord the number against the prompt hash so you have a history. A quality figure that only exists in somebody's memory of last month is not a baseline.\n\n## Prompts in the database\n\nThere is a real argument for editable prompts: non-engineers can improve them without a deploy. The cost is that your production behaviour is no longer in version control, and the reproduction problem above returns in full.\n\nThe compromise that works: prompts live in the repository as the source of truth, an interface lets people propose edits, and an edit becomes a pull request. If you must allow live editing, version every change, record the version with every response, and keep an audit trail of who changed what. Never allow a live edit that leaves no trace.\n\n## The smallest version of this lesson\n\nOne table, one row per model call, six columns: request ID, model, prompt version, parameters, retrieved document IDs, token counts. It costs an hour. Without it, every question anybody asks about your model's behaviour is answered with a shrug."},"check":{"question":"Quality drops noticeably one week with no deploy, no prompt change and no code change. Which recorded field would most directly identify the cause?","options":[{"text":"Token counts per request, showing whether prompts grew","correct":false,"tempting":"Useful for cost and for spotting context growth, but nothing changed in the prompt, so the counts would look normal."},{"text":"Latency percentiles, showing whether the provider slowed down","correct":false,"tempting":"Slower responses often accompany provider changes but do not establish that the model behind an alias was replaced."},{"text":"The pinned model snapshot identifier stored with each response","correct":true},{"text":"The retrieval index version, showing whether documents changed","correct":false,"tempting":"A genuine candidate and worth checking second, but the described change happened with no action on your side, which points at something moving upstream of you."}]},"prev":{"slug":"the-arithmetic-of-tokens-and-gpus","title":"What a prediction costs, worked out","url":"https://addaly.com/learn/cloud-and-shipping/the-arithmetic-of-tokens-and-gpus"},"next":{"slug":"training-serving-skew","title":"The feature computed two different ways","url":"https://addaly.com/learn/cloud-and-shipping/training-serving-skew"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Which model, which prompt, which version', in Shipping It. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/cloud-and-shipping/which-model-answered (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Which model, which prompt, which version","courseTitle":"Shipping It","module":{"index":6,"slug":"shipping-a-model","title":"Shipping a model, not just a web app"},"answer":null,"keyIdea":"A model system has four independently moving versions (model snapshot, prompt, decoding parameters and retrieval index) so pin each one and record all four with every response, or no complaint about last Tuesday can be investigated.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-complaint-you-cannot-investigate","runs":[{"text":"The complaint you cannot investigate"}]},{"t":"p","runs":[{"text":"A user writes: last Tuesday your assistant told me something wrong, here is a screenshot. You open the code. The prompt has been edited twice since. The provider has silently moved their model pointer to a newer snapshot. Somebody changed the temperature. The retrieval index was rebuilt on Wednesday."}]},{"t":"p","runs":[{"text":"You cannot reproduce the answer, so you cannot say whether it was fixed, and you cannot tell whether the change you are about to make would have prevented it. The deploy lesson gave you a commit SHA for code. A model system needs the same discipline over four more things that all change independently."}]},{"t":"h2","id":"the-four-versions-to-record-with-every-response","runs":[{"text":"The four versions to record with every response"}]},{"t":"ol","start":1,"items":[[{"text":"Model identity, pinned.","b":true},{"text":" Not "},{"text":"gpt-latest","code":true},{"text":" or "},{"text":"claude-latest","code":true},{"text":" or whatever the alias is called this year, but the dated snapshot. Aliases move under you, and when they move, quality changes without a deploy. Pin the version, and treat a provider’s deprecation notice as a scheduled piece of work rather than a surprise."}],[{"text":"Prompt version.","b":true},{"text":" The prompt is code. It belongs in the repository, in a file, with a hash, not in a database row somebody edits in an admin panel, and certainly not pasted in three places."}],[{"text":"Decoding parameters.","b":true},{"text":" Temperature, top-p, max tokens, stop sequences, tools available. A temperature of 0.9 makes reproduction impossible in principle; recording the seed where a provider supports one makes it possible again."}],[{"text":"Retrieval state.","b":true},{"text":" If context is fetched, record the index version and the identifiers of the documents that were actually retrieved. “The model hallucinated” and “the retriever returned the wrong document” look identical from outside and require completely different fixes."}]]},{"t":"p","runs":[{"text":"Store all of it on the row with the response, or in a log line with the request ID:"}]},{"t":"code","lang":"json","label":"JSON","text":"{\"request_id\":\"c9f2a1\",\"model\":\"provider/model-2026-04-12\",\"prompt\":\"answer_v7:9f31\",\n \"temp\":0.2,\"max_tokens\":600,\"index\":\"kb-2026-08-30\",\"docs\":[\"kb/412\",\"kb/77\"],\n \"in_tokens\":1204,\"out_tokens\":388,\"latency_ms\":2140}"},{"t":"p","runs":[{"text":"That single line answers the complaint, feeds the cost breakdown from the previous lesson, and tells you which release a regression started in."}]},{"t":"h2","id":"a-registry-for-models-you-train-yourself","runs":[{"text":"A registry, for models you train yourself"}]},{"t":"p","runs":[{"text":"If you produce your own models, the artifact is a file, and a file with no provenance is a liability. What to record, at minimum: the training data snapshot or its hash, the code commit, the hyperparameters, the evaluation numbers on the held-out set, who trained it and when."}]},{"t":"p","runs":[{"text":"MLflow is the free, self-hostable standard for this, and its model registry gives you a version number, a stage, and the run that produced it. DVC handles the data-and-artifact half in git. If both feel heavy, a directory named by date and a "},{"text":"model.json","code":true},{"text":" beside every artifact carrying those six fields is a legitimate starting point: the point is that the fields exist, not that a platform holds them."}]},{"t":"p","runs":[{"text":"One rule: the artifact you evaluated must be the artifact you deploy. Re-training “the same model” for production, with the same script and different randomness, produces a different model with unverified numbers. This is the build-once-run-anywhere argument, transferred."}]},{"t":"h2","id":"evaluate-before-you-ship-on-a-set-that-never-moves","runs":[{"text":"Evaluate before you ship, on a set that never moves"}]},{"t":"p","runs":[{"text":"This course is not the evaluation course, and the details of building a held-out set belong there. The shipping-side rule is short: a model or prompt change is a deploy, so it needs a check that runs automatically and can block it."}]},{"t":"p","runs":[{"text":"In practice that is a file of 50 to 300 examples with expected properties, run in CI, reporting a number. Not a perfect measure (nobody has one) but a tripwire. Teams without it discover regressions from users; teams with it discover them in the pull request. The gap between those two is usually one afternoon of work."}]},{"t":"p","runs":[{"text":"Record the number against the prompt hash so you have a history. A quality figure that only exists in somebody’s memory of last month is not a baseline."}]},{"t":"h2","id":"prompts-in-the-database","runs":[{"text":"Prompts in the database"}]},{"t":"p","runs":[{"text":"There is a real argument for editable prompts: non-engineers can improve them without a deploy. The cost is that your production behaviour is no longer in version control, and the reproduction problem above returns in full."}]},{"t":"p","runs":[{"text":"The compromise that works: prompts live in the repository as the source of truth, an interface lets people propose edits, and an edit becomes a pull request. If you must allow live editing, version every change, record the version with every response, and keep an audit trail of who changed what. Never allow a live edit that leaves no trace."}]},{"t":"h2","id":"the-smallest-version-of-this-lesson","runs":[{"text":"The smallest version of this lesson"}]},{"t":"p","runs":[{"text":"One table, one row per model call, six columns: request ID, model, prompt version, parameters, retrieved document IDs, token counts. It costs an hour. Without it, every question anybody asks about your model’s behaviour is answered with a shrug."}]}],"check":{"question":[{"text":"Quality drops noticeably one week with no deploy, no prompt change and no code change. Which recorded field would most directly identify the cause?"}],"options":[{"text":[{"text":"Token counts per request, showing whether prompts grew"}],"correct":false,"tempting":[{"text":"Useful for cost and for spotting context growth, but nothing changed in the prompt, so the counts would look normal."}]},{"text":[{"text":"Latency percentiles, showing whether the provider slowed down"}],"correct":false,"tempting":[{"text":"Slower responses often accompany provider changes but do not establish that the model behind an alias was replaced."}]},{"text":[{"text":"The pinned model snapshot identifier stored with each response"}],"correct":true,"tempting":null},{"text":[{"text":"The retrieval index version, showing whether documents changed"}],"correct":false,"tempting":[{"text":"A genuine candidate and worth checking second, but the described change happened with no action on your side, which points at something moving upstream of you."}]}],"order":[1,3,2,0],"why":null},"prev":{"slug":"the-arithmetic-of-tokens-and-gpus","title":"What a prediction costs, worked out","minutes":10},"next":{"slug":"training-serving-skew","title":"The feature computed two different ways","minutes":10}}}