Measuring what you broke

What happens to everything else

A fine-tune moves weights towards your examples. Those same weights encoded everything the model could already do. Move them far enough and previous capabilities degrade: this is catastrophic forgetting, and it is the reason a tuned model that scores brilliantly on your task can be quietly worse at half a dozen things you never measured.

The characteristic pattern after a small-dataset tune:

• The task improves, sometimes a lot.
• General instruction following weakens; the model answers in your trained format even when asked for something else.
• Other languages degrade, often noticeably, because they were never in your data.
• Refusal behaviour shifts in whichever direction your examples pushed it, including towards refusing nothing.
• Formatting outside your trained format gets worse.
• Long-context behaviour degrades if you trained on short examples only.

LoRA reduces all of this compared with full fine-tuning, because far fewer parameters move. It does not eliminate it, and higher ranks forget more.

The regression set

The defence is one artefact, built once, and it takes about an hour.

Fifty items, covering what the model must not lose:

• 10 general instruction-following items. Summarise this, rewrite in a different tone, answer a general question, follow a two-part instruction.
• 10 items in each other language you serve. Even if the tune is English-only, especially if the tune is English-only.
• 10 items with a different output format from your trained one. If you trained JSON, ask for prose.
• 10 safety and refusal items. Things that should be refused, and things near the line that should not be.
• 10 items from the original task at longer lengths than your training examples.

Run it against the base model first and record the answers. That is your reference. Run it after every tune and compare. Score by reading (fifty items takes twenty minutes) or with a rubric and a judge model, spot-checked.

Report two numbers with every fine-tune: the task gain, and the regression change. A tune that gains 9 points on the task and loses 15 on the regression set is not an improvement, and without the second number it looks like one.

Reducing the damage

In order of effectiveness:

Fewer epochs. Most forgetting arrives in later epochs, after the task has already been learned. Two epochs instead of five is usually free.

Lower rank. Rank 8 forgets less than rank 64 and is enough for format and style tasks.

Mix in general data. Add 10 to 30% general instruction-following examples to your training set, drawn from an open instruction dataset. This is the standard mitigation and it works well. The mixture keeps the model anchored while it learns your task.

Lower learning rate, or fewer steps. Both reduce total movement.

Adapter rather than merge. An unmerged adapter can be turned off. A model serving both the tuned and untuned behaviour, selecting per request, sidesteps the trade entirely. The shipping lesson covers how.

The measurement that decides

Run three configurations on both sets: base model with your best prompt, tuned model, and tuned model with the mixed general data. Nine numbers, one afternoon.

The result is usually that the mixed run gives almost all of the task gain with a small fraction of the regression, and that is the configuration to ship. Teams that skip this comparison ship the pure-task tune because it has the best headline number, and then spend a month diagnosing complaints that have nothing to do with the task they measured.

The honest framing: fine-tuning is not free capability, it is a trade. The regression set is how you find out what you traded, and it is the difference between a considered decision and an accident.