Retiring a fine-tune

Schedule the question

Every quarter, for every fine-tune you maintain, answer one question with numbers: does this still beat the cost-matched baseline by enough to justify what it costs to keep?

It takes an afternoon. Re-run the frozen held-out set against three things: your tuned model, the current best base model with your best prompt, and the cheapest hosted model with the same prompt. Put them on the cost table from earlier in this module.

The reason this must be scheduled is that nobody ever asks it spontaneously. A working model is not a problem, so it is never on anybody’s list, and it quietly accrues maintenance cost for years after the capability it provided became free.

The four reasons to retire

The base caught up. The most common, and the happiest. What you trained for arrives in the next generation of base models as default behaviour. The correct response is to delete the fine-tune and ship the prompt, and to record that you did, because it is evidence your team makes decisions on numbers rather than on attachment.

The task changed. The taxonomy was redrawn, the product moved, the six queues became four. A model trained on the old definition is now systematically wrong, and it is wrong confidently.

The volume collapsed. The fixed costs no longer amortise. A model serving 20,000 requests a month costs more to host than the same traffic costs on an API by a wide margin.

Nobody owns it. The person who built it left, the pipeline has not been run in a year, and no one can say what it was trained on. This is the most dangerous state in this lesson (a model still making decisions that nobody can explain or rebuild) and the correct action is to retire it even if it is performing, because you have no way to establish that it is.

Retiring gracefully

1. Stand the replacement up in shadow, serving nothing, and compare output distributions on live traffic.
2. Move a slice of traffic across and watch the user-behaviour proxies for a week.
3. Complete the switch, keeping the old artefact and its manifest for a rollback window.
4. After the window, delete the serving infrastructure.
5. Keep the decision record, the data card and the model card. They cost kilobytes and they are the institutional memory of what was tried and what it produced. The next person to propose this fine-tune should find out in ten minutes that it was built, measured and retired, and why.

Deletion, and the part that is unresolved

If you are retiring because of a data-protection obligation rather than economics, the picture is less tidy.

You can delete the dataset, the checkpoints, the merged weights, the quantised artefacts, the adapters in every registry, and the backups. That is tractable and you should have the inventory to do it, which is one more argument for the manifest.

What you cannot straightforwardly do is remove a specific person’s influence from weights that remain in service. Whether that is required is not settled: regulators have expressed different views, the question of whether model weights constitute personal data is argued seriously on both sides, and the answer differs by jurisdiction. Nobody can currently tell you with confidence what the obligation is, and this course will not pretend otherwise.

Machine unlearning (methods that approximately remove one example’s influence without full retraining) is an active research field with genuine results and genuine limits. The honest summary is that it is not yet something to build a compliance commitment on. The engineering answer that works under any interpretation is the one from the privacy lesson: keep the dataset rebuildable, retrain on a schedule, and record which data version produced which model, so that removing somebody’s data from the next build is routine.

The closing thought for the course

This course opened by arguing that most fine-tuning projects should not happen, and it ends by arguing that many of the ones that did happen should now stop. Both are the same claim.

A fine-tune is a bet that a specific capability is worth owning, maintaining and defending against a field that improves underneath it every few months. Sometimes that bet is clearly right: a latency budget, an air gap, a distillation at volume, a verifiable task where reliability is the product. Often it is not, and it was made because training a model is more interesting than writing a prompt.

The discipline that separates the two is the same at the start and at the end: a frozen baseline, a held-out set you did not choose afterwards, a number with an interval, and a stated condition under which you would stop. Everything else in these nine modules is machinery. That is the method.