The day the base model moves

Adapters do not transfer

A LoRA is a set of deltas fitted against one exact set of base weights. Apply it to different weights, even the next release in the same family, with the identical architecture, and you are perturbing matrices it was never fitted against.

The unpleasant part: it loads without error. Shapes match, the adapter attaches, the model generates. It is worse, in ways that look like a regression rather than a category error. There is no exception to catch.

The same risk exists within a version. Model repositories are edited in place: a tokenizer fix, a re-uploaded shard, a chat template correction. Pin the revision hash of the base model in your config, and record it beside the adapter, so that “the same base model” means the same bytes.

What does transfer

Everything that matters, if you kept it:

• The dataset, with its filters and its data card.
• The evaluation set, the canary set and the refusal set.
• The configuration, including the learning rate and target modules you arrived at.
• The ablation log, which tells you what does and does not help on this data.

The model was always the perishable output. These are the durable ones, and a team that has them treats a base-model release as an afternoon.

The decision at each release

When a meaningfully better base ships, which for open-weight families has historically been every six to nine months, work through four questions in order.

1. Does the new base, with a good prompt, already beat our tuned old model?

Run it. This takes an hour and it is often yes, particularly when your fine-tune was addressing instruction-following or format adherence, which is exactly what each generation of base models does better.

If yes, the correct action is to delete the fine-tune and ship the prompt. That is a success (it means the capability you paid to install is now free) and it is regularly resisted for reasons that have more to do with sunk effort than with the product.

2. Does the new base tokenise our language better? For non-English work this is often the largest single improvement in a release, and it changes cost and context as well as quality.

3. What does retraining cost us? If the pipeline is one command, this is not a real question. If nobody has run the rebuild in eight months, find out now rather than under pressure.

4. Is anything in the interface different? A new chat template, new special tokens, a different system-prompt convention. These do not break loudly; they degrade.

The rebuild target

The answer to question three should be a command:

rebuild: dataset train merge quantise evaluate
	@echo "artefact: out/merged-q4km.gguf"

Run it once a quarter against the current base whether or not you need to. A pipeline that is exercised regularly works; one that is exercised annually does not, and you discover that at the worst moment.

Time it and write the number in the decision record. “Rebuild: 4 hours, unattended” changes how an organisation reasons about its dependency on a model.

The migration itself

1. Retrain on the new base with the unchanged dataset and config.
2. Evaluate old and new on the same held-out set, canary set and refusal set.
3. Shadow the new model against live traffic without serving its output, comparing distributions of length, refusal rate and fallback rate.
4. Roll out to a slice, watch the user-behaviour proxies, then complete.
5. Keep the old artefact for a rollback window.

And re-run the cost table, because the new base may be a different size and change the arithmetic that justified self-hosting.

The framing worth keeping

A fine-tune is not a thing you make. It is a thing you maintain, on a cadence set by somebody else’s release schedule.

That is the honest closing argument of module one, arriving here as a practical matter. The projects that survive base-model churn are the ones whose durable artefacts (dataset, evaluation, configuration, decision record) were treated as the deliverable, with the weights as a build output. The projects that do not survive it are the ones where somebody produced a model.