{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"the-decision-record","title":"Writing the decision down","url":"https://addaly.com/learn/fine-tuning/the-decision-record","position":9,"of":84,"minutes":8,"answer":"One page written before any training (quantified failure, diagnosis, frozen baseline, budget with a kill criterion, a list of what must not get worse.","keyIdea":"One page written before any training (quantified failure, diagnosis, frozen baseline, budget with a kill criterion, a list of what must not get worse, and a rebuild estimate) is what makes a fine-tune's result believable or its abandonment cheap."},"body":{"format":"markdown","text":"## A page, before any GPU\n\nFine-tuning projects fail quietly. Three weeks in, nobody can remember what the original complaint was, the evaluation set has been edited twice to be kinder, and the question \"is this better than the prompt we had in March\" has no answer because the March prompt was deleted. The fix is one page written before anything runs.\n\nIt has six parts, and each one is a commitment you can be held to.\n\n## 1. The failure, quantified\n\nNot \"the model is not good enough\". Something like: *on 200 held-out support tickets, the current prompt produces valid JSON 78% of the time and routes to the correct queue 84% of the time. Target: 97% valid, 92% correct.*\n\nIf you cannot write this sentence, you do not have a project yet. You have a feeling, and the most common outcome of training on a feeling is a model that satisfies the feeling and nothing else.\n\n## 2. The diagnosis\n\nOne line naming which of the six causes you found, and the evidence. *Thirty failures sorted: 19 adherence, 6 format, 3 knowledge, 2 capability. The paste test resolved all three knowledge cases, so those go to retrieval.*\n\n## 3. The baseline that must be beaten\n\nWrite down the exact artefact you are competing against, and freeze it. Not \"the current prompt\" but the prompt file at a specific commit, with a specific model and specific sampling parameters. A moving baseline is why so many fine-tunes look like wins.\n\nList at least two:\n- **The cheap baseline.** The best prompt you can write in one more day, including few-shot examples and constrained decoding.\n- **The cost-matched baseline.** If the goal is cost, the honest comparison is the *smallest untuned model that a good prompt makes adequate*, not the frontier model you are currently paying for.\n\n## 4. The budget and the kill criteria\n\nTwo numbers and one sentence:\n\n> Budget: 10 engineer-days and $200 of compute. If after 10 days the tuned model has not beaten the frozen prompt baseline by at least 6 points on routing accuracy with a 95% interval that excludes zero, we stop and ship the prompt.\n\nThe kill criterion is the part people leave out, and it is the part that makes the record worth writing. Sunk cost is not a hypothesis. Decide the stopping rule while you are still indifferent.\n\n## 5. What must not get worse\n\nName the capabilities you are not trying to change but will be blamed for breaking: general instruction following, refusal behaviour on abusive inputs, the other language your product supports, answers in the domain you did not train on. These become the canary set, and it is written before training precisely so that you cannot choose it afterwards to flatter the result.\n\n## 6. Ownership and the rebuild test\n\nWho owns this artefact, where the dataset lives, and the answer to one question: *if the base model is superseded next quarter, how many days to rebuild?* If the answer is more than two, fix the pipeline before you train, not after.\n\n## The template\n\n```markdown\n# Fine-tune decision: ticket routing, 2026-09\n\nFailure      : 84% routing accuracy, 78% valid JSON on eval-v3 (n=200)\nTarget       : 92% / 97%\nDiagnosis    : 19/30 adherence, 6 format, 3 knowledge (→ retrieval), 2 capability\nBaseline     : prompts/route.md @ a91f2c, model X, temp 0\nCost baseline: smallest model that passes with a good prompt\nBudget       : 10 days, $200. Kill if < +6 pts with CI excluding 0.\nMust not break: refusals, Hindi inputs, the escalation path\nOwner        : ___   Dataset: git@…/routing-data   Rebuild: 1 day\n```\n\nTen lines. It costs twenty minutes and it is the only artefact in the project that is still useful if the fine-tune is abandoned, because a well-written record of *why we decided not to* is worth as much to the next person as a model would have been.\n\n## The uncomfortable part\n\nWrite the record honestly and a meaningful fraction of proposed fine-tunes will not survive it. That is the point. The most valuable outcome of this module is the projects it stops, and the second most valuable is that the ones which proceed do so with a frozen baseline, a stated interval and a stopping rule, which is to say, with the conditions under which a result could be believed."},"check":{"question":"Why does the decision record insist the baseline prompt be frozen at a specific commit with fixed sampling parameters, rather than simply described as 'our current prompt'?","options":[{"text":"Because sampling parameters change the tuned model's behaviour, so they must match between the two systems for the comparison to be valid.","correct":false,"tempting":"Correctly notes that parameters matter but misplaces the reason: the problem is the baseline moving over time, not the two systems differing in settings."},{"text":"Because git commits are required for reproducibility audits under most model governance frameworks.","correct":false,"tempting":"Reaches for a compliance justification for what is an experimental-design requirement that would hold with no governance regime at all."},{"text":"Prompts drift during a project, so an unfrozen baseline makes the tuned model's measured gain unattributable.","correct":true},{"text":"Because a frozen prompt can be shipped as a fallback if the fine-tune fails, which the kill criterion depends on.","correct":false,"tempting":"Describes a useful side effect; the fallback would still exist without freezing, and freezing is motivated by measurement, not deployment."}]},"prev":{"slug":"licences-and-what-you-may-train-on","title":"Licences, and what you may train on","url":"https://addaly.com/learn/fine-tuning/licences-and-what-you-may-train-on"},"next":{"slug":"one-training-step","title":"One training step, taken apart","url":"https://addaly.com/learn/fine-tuning/one-training-step"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Writing the decision down', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/the-decision-record (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Writing the decision down","courseTitle":"Fine-Tuning, and When Not To","module":{"index":1,"slug":"deciding-whether-to-fine-tune","title":"Deciding whether to fine-tune at all"},"answer":null,"keyIdea":"One page written before any training (quantified failure, diagnosis, frozen baseline, budget with a kill criterion, a list of what must not get worse, and a rebuild estimate) is what makes a fine-tune’s result believable or its abandonment cheap.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"a-page-before-any-gpu","runs":[{"text":"A page, before any GPU"}]},{"t":"p","runs":[{"text":"Fine-tuning projects fail quietly. Three weeks in, nobody can remember what the original complaint was, the evaluation set has been edited twice to be kinder, and the question “is this better than the prompt we had in March” has no answer because the March prompt was deleted. The fix is one page written before anything runs."}]},{"t":"p","runs":[{"text":"It has six parts, and each one is a commitment you can be held to."}]},{"t":"h2","id":"1-the-failure-quantified","runs":[{"text":"1. The failure, quantified"}]},{"t":"p","runs":[{"text":"Not “the model is not good enough”. Something like: "},{"text":"on 200 held-out support tickets, the current prompt produces valid JSON 78% of the time and routes to the correct queue 84% of the time. Target: 97% valid, 92% correct.","i":true}]},{"t":"p","runs":[{"text":"If you cannot write this sentence, you do not have a project yet. You have a feeling, and the most common outcome of training on a feeling is a model that satisfies the feeling and nothing else."}]},{"t":"h2","id":"2-the-diagnosis","runs":[{"text":"2. The diagnosis"}]},{"t":"p","runs":[{"text":"One line naming which of the six causes you found, and the evidence. "},{"text":"Thirty failures sorted: 19 adherence, 6 format, 3 knowledge, 2 capability. The paste test resolved all three knowledge cases, so those go to retrieval.","i":true}]},{"t":"h2","id":"3-the-baseline-that-must-be-beaten","runs":[{"text":"3. The baseline that must be beaten"}]},{"t":"p","runs":[{"text":"Write down the exact artefact you are competing against, and freeze it. Not “the current prompt” but the prompt file at a specific commit, with a specific model and specific sampling parameters. A moving baseline is why so many fine-tunes look like wins."}]},{"t":"p","runs":[{"text":"List at least two:"}]},{"t":"ul","items":[[{"text":"The cheap baseline.","b":true},{"text":" The best prompt you can write in one more day, including few-shot examples and constrained decoding."}],[{"text":"The cost-matched baseline.","b":true},{"text":" If the goal is cost, the honest comparison is the "},{"text":"smallest untuned model that a good prompt makes adequate","i":true},{"text":", not the frontier model you are currently paying for."}]]},{"t":"h2","id":"4-the-budget-and-the-kill-criteria","runs":[{"text":"4. The budget and the kill criteria"}]},{"t":"p","runs":[{"text":"Two numbers and one sentence:"}]},{"t":"quote","runs":[{"text":"Budget: 10 engineer-days and $200 of compute. If after 10 days the tuned model has not beaten the frozen prompt baseline by at least 6 points on routing accuracy with a 95% interval that excludes zero, we stop and ship the prompt."}]},{"t":"p","runs":[{"text":"The kill criterion is the part people leave out, and it is the part that makes the record worth writing. Sunk cost is not a hypothesis. Decide the stopping rule while you are still indifferent."}]},{"t":"h2","id":"5-what-must-not-get-worse","runs":[{"text":"5. What must not get worse"}]},{"t":"p","runs":[{"text":"Name the capabilities you are not trying to change but will be blamed for breaking: general instruction following, refusal behaviour on abusive inputs, the other language your product supports, answers in the domain you did not train on. These become the canary set, and it is written before training precisely so that you cannot choose it afterwards to flatter the result."}]},{"t":"h2","id":"6-ownership-and-the-rebuild-test","runs":[{"text":"6. Ownership and the rebuild test"}]},{"t":"p","runs":[{"text":"Who owns this artefact, where the dataset lives, and the answer to one question: "},{"text":"if the base model is superseded next quarter, how many days to rebuild?","i":true},{"text":" If the answer is more than two, fix the pipeline before you train, not after."}]},{"t":"h2","id":"the-template","runs":[{"text":"The template"}]},{"t":"code","lang":"markdown","label":"Markdown","text":"# Fine-tune decision: ticket routing, 2026-09\n\nFailure      : 84% routing accuracy, 78% valid JSON on eval-v3 (n=200)\nTarget       : 92% / 97%\nDiagnosis    : 19/30 adherence, 6 format, 3 knowledge (→ retrieval), 2 capability\nBaseline     : prompts/route.md @ a91f2c, model X, temp 0\nCost baseline: smallest model that passes with a good prompt\nBudget       : 10 days, $200. Kill if < +6 pts with CI excluding 0.\nMust not break: refusals, Hindi inputs, the escalation path\nOwner        : ___   Dataset: git@…/routing-data   Rebuild: 1 day"},{"t":"p","runs":[{"text":"Ten lines. It costs twenty minutes and it is the only artefact in the project that is still useful if the fine-tune is abandoned, because a well-written record of "},{"text":"why we decided not to","i":true},{"text":" is worth as much to the next person as a model would have been."}]},{"t":"h2","id":"the-uncomfortable-part","runs":[{"text":"The uncomfortable part"}]},{"t":"p","runs":[{"text":"Write the record honestly and a meaningful fraction of proposed fine-tunes will not survive it. That is the point. The most valuable outcome of this module is the projects it stops, and the second most valuable is that the ones which proceed do so with a frozen baseline, a stated interval and a stopping rule, which is to say, with the conditions under which a result could be believed."}]}],"check":{"question":[{"text":"Why does the decision record insist the baseline prompt be frozen at a specific commit with fixed sampling parameters, rather than simply described as ‘our current prompt’?"}],"options":[{"text":[{"text":"Because sampling parameters change the tuned model’s behaviour, so they must match between the two systems for the comparison to be valid."}],"correct":false,"tempting":[{"text":"Correctly notes that parameters matter but misplaces the reason: the problem is the baseline moving over time, not the two systems differing in settings."}]},{"text":[{"text":"Because git commits are required for reproducibility audits under most model governance frameworks."}],"correct":false,"tempting":[{"text":"Reaches for a compliance justification for what is an experimental-design requirement that would hold with no governance regime at all."}]},{"text":[{"text":"Prompts drift during a project, so an unfrozen baseline makes the tuned model’s measured gain unattributable."}],"correct":true,"tempting":null},{"text":[{"text":"Because a frozen prompt can be shipped as a fallback if the fine-tune fails, which the kill criterion depends on."}],"correct":false,"tempting":[{"text":"Describes a useful side effect; the fallback would still exist without freezing, and freezing is motivated by measurement, not deployment."}]}],"order":[1,0,3,2],"why":null},"prev":{"slug":"licences-and-what-you-may-train-on","title":"Licences, and what you may train on","minutes":10},"next":{"slug":"one-training-step","title":"One training step, taken apart","minutes":10}}}