{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"retiring-a-fine-tune","title":"Retiring a fine-tune","url":"https://addaly.com/learn/fine-tuning/retiring-a-fine-tune","position":84,"of":84,"minutes":9,"answer":"Schedule the quarterly question of whether a fine-tune still beats its cost-matched baseline, retire it happily when the base model catches up, keep the decision record and cards after the weights are deleted.","keyIdea":"Schedule the quarterly question of whether a fine-tune still beats its cost-matched baseline, retire it happily when the base model catches up, keep the decision record and cards after the weights are deleted, and treat a model nobody can rebuild as one to retire even while it is performing."},"body":{"format":"markdown","text":"## Schedule the question\n\nEvery quarter, for every fine-tune you maintain, answer one question with numbers: *does this still beat the cost-matched baseline by enough to justify what it costs to keep?*\n\nIt takes an afternoon. Re-run the frozen held-out set against three things: your tuned model, the current best base model with your best prompt, and the cheapest hosted model with the same prompt. Put them on the cost table from earlier in this module.\n\nThe reason this must be scheduled is that nobody ever asks it spontaneously. A working model is not a problem, so it is never on anybody's list, and it quietly accrues maintenance cost for years after the capability it provided became free.\n\n## The four reasons to retire\n\n**The base caught up.** The most common, and the happiest. What you trained for arrives in the next generation of base models as default behaviour. The correct response is to delete the fine-tune and ship the prompt, and to record that you did, because it is evidence your team makes decisions on numbers rather than on attachment.\n\n**The task changed.** The taxonomy was redrawn, the product moved, the six queues became four. A model trained on the old definition is now systematically wrong, and it is wrong confidently.\n\n**The volume collapsed.** The fixed costs no longer amortise. A model serving 20,000 requests a month costs more to host than the same traffic costs on an API by a wide margin.\n\n**Nobody owns it.** The person who built it left, the pipeline has not been run in a year, and no one can say what it was trained on. This is the most dangerous state in this lesson (a model still making decisions that nobody can explain or rebuild) and the correct action is to retire it even if it is performing, because you have no way to establish that it is.\n\n## Retiring gracefully\n\n1. Stand the replacement up in shadow, serving nothing, and compare output distributions on live traffic.\n2. Move a slice of traffic across and watch the user-behaviour proxies for a week.\n3. Complete the switch, keeping the old artefact and its manifest for a rollback window.\n4. After the window, delete the serving infrastructure.\n5. **Keep the decision record, the data card and the model card.** They cost kilobytes and they are the institutional memory of what was tried and what it produced. The next person to propose this fine-tune should find out in ten minutes that it was built, measured and retired, and why.\n\n## Deletion, and the part that is unresolved\n\nIf you are retiring because of a data-protection obligation rather than economics, the picture is less tidy.\n\nYou can delete the dataset, the checkpoints, the merged weights, the quantised artefacts, the adapters in every registry, and the backups. That is tractable and you should have the inventory to do it, which is one more argument for the manifest.\n\nWhat you cannot straightforwardly do is remove a specific person's influence from weights that remain in service. Whether that is required is not settled: regulators have expressed different views, the question of whether model weights constitute personal data is argued seriously on both sides, and the answer differs by jurisdiction. Nobody can currently tell you with confidence what the obligation is, and this course will not pretend otherwise.\n\n**Machine unlearning** (methods that approximately remove one example's influence without full retraining) is an active research field with genuine results and genuine limits. The honest summary is that it is not yet something to build a compliance commitment on. The engineering answer that works under any interpretation is the one from the privacy lesson: keep the dataset rebuildable, retrain on a schedule, and record which data version produced which model, so that removing somebody's data from the next build is routine.\n\n## The closing thought for the course\n\nThis course opened by arguing that most fine-tuning projects should not happen, and it ends by arguing that many of the ones that did happen should now stop. Both are the same claim.\n\nA fine-tune is a bet that a specific capability is worth owning, maintaining and defending against a field that improves underneath it every few months. Sometimes that bet is clearly right: a latency budget, an air gap, a distillation at volume, a verifiable task where reliability is the product. Often it is not, and it was made because training a model is more interesting than writing a prompt.\n\nThe discipline that separates the two is the same at the start and at the end: a frozen baseline, a held-out set you did not choose afterwards, a number with an interval, and a stated condition under which you would stop. Everything else in these nine modules is machinery. That is the method."},"check":{"question":"A fine-tune still performs well, but the engineer who built it has left, the pipeline has not been run in a year, and nobody can say what it was trained on. Why is retiring it the right action despite the performance?","options":[{"text":"Unmaintained models degrade over time as their weights drift, so performance will fall regardless.","correct":false,"tempting":"Attributes a property to the weights that they do not have; stored weights are fixed and do not drift on their own."},{"text":"The licence of the base model will eventually require re-acceptance, which nobody remaining is authorised to do.","correct":false,"tempting":"Invents a renewal obligation that model licences do not impose on already-trained derivatives."},{"text":"Models older than a year are contaminated relative to current benchmarks, making any reported evaluation invalid.","correct":false,"tempting":"Misapplies contamination, which concerns overlap between training data and evaluation items rather than the age of a model."},{"text":"Its performance cannot be established or explained, and it cannot be rebuilt.","correct":true}]},"prev":{"slug":"disclosure-and-model-cards","title":"Model cards and disclosure","url":"https://addaly.com/learn/fine-tuning/disclosure-and-model-cards"},"next":null,"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Retiring a fine-tune', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/retiring-a-fine-tune (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Retiring a fine-tune","courseTitle":"Fine-Tuning, and When Not To","module":{"index":9,"slug":"shipping-and-living-with-it","title":"Shipping it, and living with it"},"answer":null,"keyIdea":"Schedule the quarterly question of whether a fine-tune still beats its cost-matched baseline, retire it happily when the base model catches up, keep the decision record and cards after the weights are deleted, and treat a model nobody can rebuild as one to retire even while it is performing.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"schedule-the-question","runs":[{"text":"Schedule the question"}]},{"t":"p","runs":[{"text":"Every quarter, for every fine-tune you maintain, answer one question with numbers: "},{"text":"does this still beat the cost-matched baseline by enough to justify what it costs to keep?","i":true}]},{"t":"p","runs":[{"text":"It takes an afternoon. Re-run the frozen held-out set against three things: your tuned model, the current best base model with your best prompt, and the cheapest hosted model with the same prompt. Put them on the cost table from earlier in this module."}]},{"t":"p","runs":[{"text":"The reason this must be scheduled is that nobody ever asks it spontaneously. A working model is not a problem, so it is never on anybody’s list, and it quietly accrues maintenance cost for years after the capability it provided became free."}]},{"t":"h2","id":"the-four-reasons-to-retire","runs":[{"text":"The four reasons to retire"}]},{"t":"p","runs":[{"text":"The base caught up.","b":true},{"text":" The most common, and the happiest. What you trained for arrives in the next generation of base models as default behaviour. The correct response is to delete the fine-tune and ship the prompt, and to record that you did, because it is evidence your team makes decisions on numbers rather than on attachment."}]},{"t":"p","runs":[{"text":"The task changed.","b":true},{"text":" The taxonomy was redrawn, the product moved, the six queues became four. A model trained on the old definition is now systematically wrong, and it is wrong confidently."}]},{"t":"p","runs":[{"text":"The volume collapsed.","b":true},{"text":" The fixed costs no longer amortise. A model serving 20,000 requests a month costs more to host than the same traffic costs on an API by a wide margin."}]},{"t":"p","runs":[{"text":"Nobody owns it.","b":true},{"text":" The person who built it left, the pipeline has not been run in a year, and no one can say what it was trained on. This is the most dangerous state in this lesson (a model still making decisions that nobody can explain or rebuild) and the correct action is to retire it even if it is performing, because you have no way to establish that it is."}]},{"t":"h2","id":"retiring-gracefully","runs":[{"text":"Retiring gracefully"}]},{"t":"ol","start":1,"items":[[{"text":"Stand the replacement up in shadow, serving nothing, and compare output distributions on live traffic."}],[{"text":"Move a slice of traffic across and watch the user-behaviour proxies for a week."}],[{"text":"Complete the switch, keeping the old artefact and its manifest for a rollback window."}],[{"text":"After the window, delete the serving infrastructure."}],[{"text":"Keep the decision record, the data card and the model card.","b":true},{"text":" They cost kilobytes and they are the institutional memory of what was tried and what it produced. The next person to propose this fine-tune should find out in ten minutes that it was built, measured and retired, and why."}]]},{"t":"h2","id":"deletion-and-the-part-that-is-unresolved","runs":[{"text":"Deletion, and the part that is unresolved"}]},{"t":"p","runs":[{"text":"If you are retiring because of a data-protection obligation rather than economics, the picture is less tidy."}]},{"t":"p","runs":[{"text":"You can delete the dataset, the checkpoints, the merged weights, the quantised artefacts, the adapters in every registry, and the backups. That is tractable and you should have the inventory to do it, which is one more argument for the manifest."}]},{"t":"p","runs":[{"text":"What you cannot straightforwardly do is remove a specific person’s influence from weights that remain in service. Whether that is required is not settled: regulators have expressed different views, the question of whether model weights constitute personal data is argued seriously on both sides, and the answer differs by jurisdiction. Nobody can currently tell you with confidence what the obligation is, and this course will not pretend otherwise."}]},{"t":"p","runs":[{"text":"Machine unlearning","b":true},{"text":" (methods that approximately remove one example’s influence without full retraining) is an active research field with genuine results and genuine limits. The honest summary is that it is not yet something to build a compliance commitment on. The engineering answer that works under any interpretation is the one from the privacy lesson: keep the dataset rebuildable, retrain on a schedule, and record which data version produced which model, so that removing somebody’s data from the next build is routine."}]},{"t":"h2","id":"the-closing-thought-for-the-course","runs":[{"text":"The closing thought for the course"}]},{"t":"p","runs":[{"text":"This course opened by arguing that most fine-tuning projects should not happen, and it ends by arguing that many of the ones that did happen should now stop. Both are the same claim."}]},{"t":"p","runs":[{"text":"A fine-tune is a bet that a specific capability is worth owning, maintaining and defending against a field that improves underneath it every few months. Sometimes that bet is clearly right: a latency budget, an air gap, a distillation at volume, a verifiable task where reliability is the product. Often it is not, and it was made because training a model is more interesting than writing a prompt."}]},{"t":"p","runs":[{"text":"The discipline that separates the two is the same at the start and at the end: a frozen baseline, a held-out set you did not choose afterwards, a number with an interval, and a stated condition under which you would stop. Everything else in these nine modules is machinery. That is the method."}]}],"check":{"question":[{"text":"A fine-tune still performs well, but the engineer who built it has left, the pipeline has not been run in a year, and nobody can say what it was trained on. Why is retiring it the right action despite the performance?"}],"options":[{"text":[{"text":"Unmaintained models degrade over time as their weights drift, so performance will fall regardless."}],"correct":false,"tempting":[{"text":"Attributes a property to the weights that they do not have; stored weights are fixed and do not drift on their own."}]},{"text":[{"text":"The licence of the base model will eventually require re-acceptance, which nobody remaining is authorised to do."}],"correct":false,"tempting":[{"text":"Invents a renewal obligation that model licences do not impose on already-trained derivatives."}]},{"text":[{"text":"Models older than a year are contaminated relative to current benchmarks, making any reported evaluation invalid."}],"correct":false,"tempting":[{"text":"Misapplies contamination, which concerns overlap between training data and evaluation items rather than the age of a model."}]},{"text":[{"text":"Its performance cannot be established or explained, and it cannot be rebuilt."}],"correct":true,"tempting":null}],"order":[0,1,3,2],"why":null},"prev":{"slug":"disclosure-and-model-cards","title":"Model cards and disclosure","minutes":9},"next":null}}