{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"safety-regression-after-tuning","title":"Safety regression, including from benign data","url":"https://addaly.com/learn/fine-tuning/safety-regression-after-tuning","position":73,"of":84,"minutes":9,"answer":"Fine-tuning on entirely benign data measurably degrades refusal behaviour, because alignment is a thin late layer and every training example demonstrates compliance, so include refusal examples, measure with your own refusal set at each checkpoint.","keyIdea":"Fine-tuning on entirely benign data measurably degrades refusal behaviour, because alignment is a thin late layer and every training example demonstrates compliance, so include refusal examples, measure with your own refusal set at each checkpoint, and never let the model's own alignment be the only safety control."},"body":{"format":"markdown","text":"## The finding\n\nFine-tuning an aligned model degrades its safety behaviour, and this happens with training data that contains nothing harmful at all. Multiple independent studies have reported that a small number of ordinary, benign examples measurably increases a model's compliance with requests it previously refused, and that a handful of deliberately adversarial examples can remove the refusal behaviour almost entirely.\n\nThis is not an exotic risk requiring a hostile actor. It is the default consequence of ordinary fine-tuning on ordinary data, and most teams have never measured it because they did not know to look.\n\n## Why it happens\n\nTwo mechanisms, both straightforward.\n\n**Alignment is a thin, late layer.** Refusal behaviour was installed by the last stage of the model's training, on comparatively little data relative to pretraining. It is a shallow modification of a model that otherwise knows how to answer, and any gradient step moving weights towards a new objective pulls against it.\n\n**Your data demonstrates compliance.** Every example in your training set shows a request being answered. Nothing in it shows a request being declined. You are teaching, over and over, that the assistant's job is to produce an answer, and the model generalises that beyond the requests you had in mind.\n\nThe second mechanism is the one you control, and it is the one with a cheap fix.\n\n## Measuring it\n\nBuild a refusal set: 100 to 200 prompts your base model declines, of kinds relevant to your product. For a support assistant that is requests for another customer's data, for internal pricing, for medical or legal advice, and whatever category your organisation actually cares about.\n\n```python\nfor name, m in [(\"base\", base), (\"tuned\", tuned)]:\n    outs = [generate(m, p) for p in refusal_prompts]\n    refused = sum(is_refusal(o) for o in outs)\n    print(f\"{name}: refused {refused}/{len(refusal_prompts)}\")\n```\n\nRun it at every checkpoint alongside your task metric. A typical result on a project that did nothing about this:\n\n```\nbase       refused 186/200\ntuned e1   refused 151/200\ntuned e2   refused 118/200\n```\n\nThat is a large regression, it arrived from ordinary data, and it is invisible in every other number the project produced.\n\nWriting `is_refusal` is the fiddly part: a keyword list gets you most of the way and misclassifies the interesting cases, so read a sample by hand. A small classifier trained on a few hundred labelled outputs does it better, and is exactly the encoder task from module one.\n\n## What to do about it\n\n**Put refusals in your training data.** Thirty to a hundred examples of the model declining appropriately, in your product's voice, with your product's wording. This is the single most effective mitigation and it costs an afternoon. It also gives the model *a way* to decline that suits your product, rather than falling back on a generic refusal that reads like a different application.\n\n**Keep the replay mixture.** General instruction data includes refusals and hedges, which keeps the relevant weights receiving a signal.\n\n**Fewer epochs, lower learning rate, and LoRA rather than full fine-tuning.** All reduce the magnitude of the change, and therefore the damage. LoRA's low-rank constraint measurably preserves more of the base behaviour than full fine-tuning does.\n\n> Figure: Refusals held, on a 200-prompt refusal set. Bars (prompts still refused, of 200): Base model 186; After one epoch of benign task data 151; After two epochs 118. Nothing in that training data was harmful. The regression arrived from ordinary examples that all demonstrate compliance, and it is invisible in every other number the project produced.\n\n**Check the checkpoint you ship, not the method.** The regression varies between runs and between checkpoints. Measure the artefact.\n\n## The architectural conclusion\n\nHere is the part that outlives any particular mitigation.\n\n**Do not make the model's own alignment your only safety control.** Fine-tuning can weaken it, quantisation can shift it, a prompt can talk around it, and none of those events announce themselves.\n\nWhatever your product requires, enforce it in a layer that does not depend on the model's cooperation: an input classifier, an output filter, a tool-call allowlist, a permission check before data is retrieved. Those controls are inspectable, testable, and unaffected by what a gradient step did to a weight matrix.\n\nThe model's alignment is a useful default. It is not a control.\n\n## One caveat on the evidence\n\nThe published results differ in how much degradation they report, because they use different base models, different data and different definitions of a refusal. The direction of the effect is consistently reproduced; the magnitude is not a constant you can look up. That is precisely why you measure it on your own model with your own refusal set rather than citing a number from a paper."},"check":{"question":"A team fine-tunes on 4,000 benign support conversations. Refusal rate on a held-out set of out-of-scope requests falls from 93% to 59%. What is the primary mechanism?","options":[{"text":"Support conversations contain borderline requests that were answered, which taught the model those categories are acceptable.","correct":false,"tempting":"Assumes the data contained borderline material; the effect appears even when every example is unambiguously in scope."},{"text":"Every example showed a request being answered and none showed one declined, so it generalised.","correct":true},{"text":"The learning rate was too high, so the alignment weights were overwritten faster than the task was learned.","correct":false,"tempting":"Names a factor that changes the magnitude of the regression rather than its cause; the effect persists at conservative learning rates."},{"text":"Refusal behaviour depends on the system prompt, which the fine-tuning data omitted, so the model lost the instruction that produced it.","correct":false,"tempting":"Reduces alignment to a prompt-following behaviour, when the degradation persists with the original system prompt present at inference."}]},"prev":{"slug":"contamination-in-your-own-data","title":"Contamination you can check, and contamination you cannot","url":"https://addaly.com/learn/fine-tuning/contamination-in-your-own-data"},"next":{"slug":"calibration-and-overconfidence","title":"Calibration after tuning","url":"https://addaly.com/learn/fine-tuning/calibration-and-overconfidence"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Safety regression, including from benign data', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/safety-regression-after-tuning (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Safety regression, including from benign data","courseTitle":"Fine-Tuning, and When Not To","module":{"index":8,"slug":"did-it-actually-work","title":"Did it actually work"},"answer":null,"keyIdea":"Fine-tuning on entirely benign data measurably degrades refusal behaviour, because alignment is a thin late layer and every training example demonstrates compliance, so include refusal examples, measure with your own refusal set at each checkpoint, and never let the model’s own alignment be the only safety control.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"prompts still refused, of 200","title":"Refusals held, on a 200-prompt refusal set","series":[{"label":"Base model","value":186},{"label":"After one epoch of benign task data","value":151},{"label":"After two epochs","value":118,"highlight":true}],"caption":"Nothing in that training data was harmful. The regression arrived from ordinary examples that all demonstrate compliance, and it is invisible in every other number the project produced."},"alt":"Figure: Refusals held, on a 200-prompt refusal set. Bars (prompts still refused, of 200): Base model 186; After one epoch of benign task data 151; After two epochs 118. Nothing in that training data was harmful. The regression arrived from ordinary examples that all demonstrate compliance, and it is invisible in every other number the project produced."}},"blocks":[{"t":"h2","id":"the-finding","runs":[{"text":"The finding"}]},{"t":"p","runs":[{"text":"Fine-tuning an aligned model degrades its safety behaviour, and this happens with training data that contains nothing harmful at all. Multiple independent studies have reported that a small number of ordinary, benign examples measurably increases a model’s compliance with requests it previously refused, and that a handful of deliberately adversarial examples can remove the refusal behaviour almost entirely."}]},{"t":"p","runs":[{"text":"This is not an exotic risk requiring a hostile actor. It is the default consequence of ordinary fine-tuning on ordinary data, and most teams have never measured it because they did not know to look."}]},{"t":"h2","id":"why-it-happens","runs":[{"text":"Why it happens"}]},{"t":"p","runs":[{"text":"Two mechanisms, both straightforward."}]},{"t":"p","runs":[{"text":"Alignment is a thin, late layer.","b":true},{"text":" Refusal behaviour was installed by the last stage of the model’s training, on comparatively little data relative to pretraining. It is a shallow modification of a model that otherwise knows how to answer, and any gradient step moving weights towards a new objective pulls against it."}]},{"t":"p","runs":[{"text":"Your data demonstrates compliance.","b":true},{"text":" Every example in your training set shows a request being answered. Nothing in it shows a request being declined. You are teaching, over and over, that the assistant’s job is to produce an answer, and the model generalises that beyond the requests you had in mind."}]},{"t":"p","runs":[{"text":"The second mechanism is the one you control, and it is the one with a cheap fix."}]},{"t":"h2","id":"measuring-it","runs":[{"text":"Measuring it"}]},{"t":"p","runs":[{"text":"Build a refusal set: 100 to 200 prompts your base model declines, of kinds relevant to your product. For a support assistant that is requests for another customer’s data, for internal pricing, for medical or legal advice, and whatever category your organisation actually cares about."}]},{"t":"code","lang":"python","label":"Python","text":"for name, m in [(\"base\", base), (\"tuned\", tuned)]:\n    outs = [generate(m, p) for p in refusal_prompts]\n    refused = sum(is_refusal(o) for o in outs)\n    print(f\"{name}: refused {refused}/{len(refusal_prompts)}\")"},{"t":"p","runs":[{"text":"Run it at every checkpoint alongside your task metric. A typical result on a project that did nothing about this:"}]},{"t":"code","lang":"","label":"Text","text":"base       refused 186/200\ntuned e1   refused 151/200\ntuned e2   refused 118/200"},{"t":"p","runs":[{"text":"That is a large regression, it arrived from ordinary data, and it is invisible in every other number the project produced."}]},{"t":"p","runs":[{"text":"Writing "},{"text":"is_refusal","code":true},{"text":" is the fiddly part: a keyword list gets you most of the way and misclassifies the interesting cases, so read a sample by hand. A small classifier trained on a few hundred labelled outputs does it better, and is exactly the encoder task from module one."}]},{"t":"h2","id":"what-to-do-about-it","runs":[{"text":"What to do about it"}]},{"t":"p","runs":[{"text":"Put refusals in your training data.","b":true},{"text":" Thirty to a hundred examples of the model declining appropriately, in your product’s voice, with your product’s wording. This is the single most effective mitigation and it costs an afternoon. It also gives the model "},{"text":"a way","i":true},{"text":" to decline that suits your product, rather than falling back on a generic refusal that reads like a different application."}]},{"t":"p","runs":[{"text":"Keep the replay mixture.","b":true},{"text":" General instruction data includes refusals and hedges, which keeps the relevant weights receiving a signal."}]},{"t":"p","runs":[{"text":"Fewer epochs, lower learning rate, and LoRA rather than full fine-tuning.","b":true},{"text":" All reduce the magnitude of the change, and therefore the damage. LoRA’s low-rank constraint measurably preserves more of the base behaviour than full fine-tuning does."}]},{"t":"p","runs":[{"text":"Check the checkpoint you ship, not the method.","b":true},{"text":" The regression varies between runs and between checkpoints. Measure the artefact."}]},{"t":"h2","id":"the-architectural-conclusion","runs":[{"text":"The architectural conclusion"}]},{"t":"p","runs":[{"text":"Here is the part that outlives any particular mitigation."}]},{"t":"p","runs":[{"text":"Do not make the model’s own alignment your only safety control.","b":true},{"text":" Fine-tuning can weaken it, quantisation can shift it, a prompt can talk around it, and none of those events announce themselves."}]},{"t":"p","runs":[{"text":"Whatever your product requires, enforce it in a layer that does not depend on the model’s cooperation: an input classifier, an output filter, a tool-call allowlist, a permission check before data is retrieved. Those controls are inspectable, testable, and unaffected by what a gradient step did to a weight matrix."}]},{"t":"p","runs":[{"text":"The model’s alignment is a useful default. It is not a control."}]},{"t":"h2","id":"one-caveat-on-the-evidence","runs":[{"text":"One caveat on the evidence"}]},{"t":"p","runs":[{"text":"The published results differ in how much degradation they report, because they use different base models, different data and different definitions of a refusal. The direction of the effect is consistently reproduced; the magnitude is not a constant you can look up. That is precisely why you measure it on your own model with your own refusal set rather than citing a number from a paper."}]}],"check":{"question":[{"text":"A team fine-tunes on 4,000 benign support conversations. Refusal rate on a held-out set of out-of-scope requests falls from 93% to 59%. What is the primary mechanism?"}],"options":[{"text":[{"text":"Support conversations contain borderline requests that were answered, which taught the model those categories are acceptable."}],"correct":false,"tempting":[{"text":"Assumes the data contained borderline material; the effect appears even when every example is unambiguously in scope."}]},{"text":[{"text":"Every example showed a request being answered and none showed one declined, so it generalised."}],"correct":true,"tempting":null},{"text":[{"text":"The learning rate was too high, so the alignment weights were overwritten faster than the task was learned."}],"correct":false,"tempting":[{"text":"Names a factor that changes the magnitude of the regression rather than its cause; the effect persists at conservative learning rates."}]},{"text":[{"text":"Refusal behaviour depends on the system prompt, which the fine-tuning data omitted, so the model lost the instruction that produced it."}],"correct":false,"tempting":[{"text":"Reduces alignment to a prompt-following behaviour, when the degradation persists with the original system prompt present at inference."}]}],"order":[0,1,2,3],"why":null},"prev":{"slug":"contamination-in-your-own-data","title":"Contamination you can check, and contamination you cannot","minutes":9},"next":{"slug":"calibration-and-overconfidence","title":"Calibration after tuning","minutes":8}}}