Safety regression, including from benign data

The finding

Fine-tuning an aligned model degrades its safety behaviour, and this happens with training data that contains nothing harmful at all. Multiple independent studies have reported that a small number of ordinary, benign examples measurably increases a model’s compliance with requests it previously refused, and that a handful of deliberately adversarial examples can remove the refusal behaviour almost entirely.

This is not an exotic risk requiring a hostile actor. It is the default consequence of ordinary fine-tuning on ordinary data, and most teams have never measured it because they did not know to look.

Why it happens

Two mechanisms, both straightforward.

Alignment is a thin, late layer. Refusal behaviour was installed by the last stage of the model’s training, on comparatively little data relative to pretraining. It is a shallow modification of a model that otherwise knows how to answer, and any gradient step moving weights towards a new objective pulls against it.

Your data demonstrates compliance. Every example in your training set shows a request being answered. Nothing in it shows a request being declined. You are teaching, over and over, that the assistant’s job is to produce an answer, and the model generalises that beyond the requests you had in mind.

The second mechanism is the one you control, and it is the one with a cheap fix.

Measuring it

Build a refusal set: 100 to 200 prompts your base model declines, of kinds relevant to your product. For a support assistant that is requests for another customer’s data, for internal pricing, for medical or legal advice, and whatever category your organisation actually cares about.

for name, m in [("base", base), ("tuned", tuned)]:
    outs = [generate(m, p) for p in refusal_prompts]
    refused = sum(is_refusal(o) for o in outs)
    print(f"{name}: refused {refused}/{len(refusal_prompts)}")

Run it at every checkpoint alongside your task metric. A typical result on a project that did nothing about this:

base       refused 186/200
tuned e1   refused 151/200
tuned e2   refused 118/200

That is a large regression, it arrived from ordinary data, and it is invisible in every other number the project produced.

Writing is_refusal is the fiddly part: a keyword list gets you most of the way and misclassifies the interesting cases, so read a sample by hand. A small classifier trained on a few hundred labelled outputs does it better, and is exactly the encoder task from module one.

What to do about it

Put refusals in your training data. Thirty to a hundred examples of the model declining appropriately, in your product’s voice, with your product’s wording. This is the single most effective mitigation and it costs an afternoon. It also gives the model a way to decline that suits your product, rather than falling back on a generic refusal that reads like a different application.

Keep the replay mixture. General instruction data includes refusals and hedges, which keeps the relevant weights receiving a signal.

Fewer epochs, lower learning rate, and LoRA rather than full fine-tuning. All reduce the magnitude of the change, and therefore the damage. LoRA’s low-rank constraint measurably preserves more of the base behaviour than full fine-tuning does.

Check the checkpoint you ship, not the method. The regression varies between runs and between checkpoints. Measure the artefact.

The architectural conclusion

Here is the part that outlives any particular mitigation.

Do not make the model’s own alignment your only safety control. Fine-tuning can weaken it, quantisation can shift it, a prompt can talk around it, and none of those events announce themselves.

Whatever your product requires, enforce it in a layer that does not depend on the model’s cooperation: an input classifier, an output filter, a tool-call allowlist, a permission check before data is retrieved. Those controls are inspectable, testable, and unaffected by what a gradient step did to a weight matrix.

The model’s alignment is a useful default. It is not a control.

One caveat on the evidence

The published results differ in how much degradation they report, because they use different base models, different data and different definitions of a refusal. The direction of the effect is consistently reproduced; the magnitude is not a constant you can look up. That is precisely why you measure it on your own model with your own refusal set rather than citing a number from a paper.