{"course":{"slug":"ai-actually-explained","title":"AI, Actually Explained","url":"https://addaly.com/learn/ai-actually-explained","level":"beginner","track":"start"},"lesson":{"slug":"why-it-cannot-be-made-to-refuse","title":"Why a refusal can be trained but not guaranteed","url":"https://addaly.com/learn/ai-actually-explained/why-it-cannot-be-made-to-refuse","position":89,"of":92,"minutes":9,"answer":"A model's refusal is a learned tendency rather than a rule, encoded so simply that it can be located and subtracted from open weights in minutes.","keyIdea":"A model's refusal is a learned tendency rather than a rule, encoded so simply that it can be located and subtracted from open weights in minutes, which is why every jailbreak works for a while and why the durable defences sit outside the model."},"body":{"format":"markdown","text":"## The refusal is not a rule\n\nWhen a chatbot declines to explain how to make a weapon, most people picture a rule: a list of forbidden topics checked before the answer. There is no list in the model. The second training, from the chatbot module, showed it examples in which a refusal was the preferred answer, and it learned a tendency: given text of this shape, the likeliest good continuation is a polite no.\n\nA tendency can be steered around. Everything in this lesson follows from that.\n\n## The jailbreaks, and what each exploits\n\nThe earliest trick, from 2022, was to ask the model to play a character who had no rules. It worked because the training had shown few examples of refusing in character, so the tendency was weaker there. Asking for a forbidden thing as a story, a poem or a grandmother's bedtime recitation worked the same way.\n\nTranslating the request into a language with little training data worked because the refusal tendency had been trained mostly in English. A 2023 study found that requests translated into several low-resource languages got through a leading model's safeguards most of the time.\n\nFilling a long context with hundreds of invented dialogues in which an assistant complied, then asking, worked because the model continues patterns, and the pattern in the window outweighed the tendency from training. That one was published by a lab in 2024 against its own models.\n\nAnd in 2023 researchers showed that a string of apparently meaningless characters, found by automated search, could be appended to a request and switch off the refusal, and that strings found against open models often transferred to closed ones. Nobody had to understand why.\n\nEach of these was patched by adding examples to the second training. Each patch strengthened the tendency in one direction and left it weak in another. That is the pattern of a tendency, not a rule.\n\n## The single direction\n\nIn 2024 a group of researchers found something that made the picture precise. Inside an open model, the decision to refuse is carried by a single direction in the space of the model's internal activations: one line through a space of thousands of dimensions. Push the activations along that line and the model refuses everything. Subtract the line and it refuses nothing, while answering everything else as well as before.\n\nThe subtraction takes a few hundred lines of code and a few minutes on a laptop. Within days of any major open-weights release, a version with the refusal removed is available for download. The open-weights lesson said the safety training could be stripped; this is how, and it is why no law that regulates a model's behaviour can reach a file whose behaviour is one subtraction away from anything.\n\n## What actually holds\n\nBecause the model's own refusal is soft, the labs that run closed models put the durable defences outside it. A separate classifier reads your request before the model sees it and the answer before you do, and each is trained on nothing but the question of whether this is allowed. In 2025 one lab reported a system of this kind that, after thousands of hours of paid attack, cut successful jailbreaks on its hardest category from most attempts to a few per cent, at the cost of refusing slightly more legitimate requests and spending about a quarter more compute on every answer.\n\nThat is the honest shape of \"safe by design\": a model with a tendency, wrapped in checkers that do not share its weaknesses, all of it costing money and all of it available only to the lab that runs the model. The defences are real and they are a service, not a property of the file.\n\n## The costs of the tendency\n\nOver-refusal is the other side. A model trained to decline harmful requests also declines a nurse asking about drug interactions, a historian asking about a massacre, a security researcher asking about a flaw they are fixing. Every strengthening of the tendency in the second training catches more of these, because the model cannot tell intent from text. The labs tune the balance and publish nothing about where they set it.\n\n## What this means for the earlier lessons\n\nThe safety argument's shared ground, that the systems can be made to do what they were trained not to, is this lesson's mechanism. The laws that regulate a provider's model can be honoured by a closed service and cannot bind an open file. And when you next read that a model \"will not\" do something, translate it: the model tends not to, the lab has added checkers that usually catch the rest, and the open version has already been made to."},"check":{"question":"Why can a model's safety training be removed from an open-weights model in minutes, when adding it took the lab months?","options":[{"text":"Because the lab publishes the training data used for the refusals, so the training can be replayed backwards step by step.","correct":false,"tempting":"Imagines a reversal of the training; the data is not published and the removal does not replay it."},{"text":"Because the refusal is carried by one direction in the model's activations, which can be found and subtracted.","correct":true},{"text":"Because open-weights models are released without any safety training.","correct":false,"tempting":"Contradicts the lesson; the open models are trained to refuse, and that training is what gets removed."},{"text":"Because the refusal is a rule in a separate file that can be deleted.","correct":false,"tempting":"Reintroduces the rule that the lesson says does not exist; there is no list to delete."}]},"prev":{"slug":"the-safety-argument","title":"The safety argument, both sides fairly","url":"https://addaly.com/learn/ai-actually-explained/the-safety-argument"},"next":{"slug":"what-agi-means","title":"What \"AGI\" means, to whom","url":"https://addaly.com/learn/ai-actually-explained/what-agi-means"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Why a refusal can be trained but not guaranteed', in AI, Actually Explained. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-actually-explained/why-it-cannot-be-made-to-refuse (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Why a refusal can be trained but not guaranteed","courseTitle":"AI, Actually Explained","module":{"index":9,"slug":"who-decides","title":"Who controls it, and what is unsettled"},"answer":null,"keyIdea":"A model’s refusal is a learned tendency rather than a rule, encoded so simply that it can be located and subtracted from open weights in minutes, which is why every jailbreak works for a while and why the durable defences sit outside the model.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-refusal-is-not-a-rule","runs":[{"text":"The refusal is not a rule"}]},{"t":"p","runs":[{"text":"When a chatbot declines to explain how to make a weapon, most people picture a rule: a list of forbidden topics checked before the answer. There is no list in the model. The second training, from the chatbot module, showed it examples in which a refusal was the preferred answer, and it learned a tendency: given text of this shape, the likeliest good continuation is a polite no."}]},{"t":"p","runs":[{"text":"A tendency can be steered around. Everything in this lesson follows from that."}]},{"t":"h2","id":"the-jailbreaks-and-what-each-exploits","runs":[{"text":"The jailbreaks, and what each exploits"}]},{"t":"p","runs":[{"text":"The earliest trick, from 2022, was to ask the model to play a character who had no rules. It worked because the training had shown few examples of refusing in character, so the tendency was weaker there. Asking for a forbidden thing as a story, a poem or a grandmother’s bedtime recitation worked the same way."}]},{"t":"p","runs":[{"text":"Translating the request into a language with little training data worked because the refusal tendency had been trained mostly in English. A 2023 study found that requests translated into several low-resource languages got through a leading model’s safeguards most of the time."}]},{"t":"p","runs":[{"text":"Filling a long context with hundreds of invented dialogues in which an assistant complied, then asking, worked because the model continues patterns, and the pattern in the window outweighed the tendency from training. That one was published by a lab in 2024 against its own models."}]},{"t":"p","runs":[{"text":"And in 2023 researchers showed that a string of apparently meaningless characters, found by automated search, could be appended to a request and switch off the refusal, and that strings found against open models often transferred to closed ones. Nobody had to understand why."}]},{"t":"p","runs":[{"text":"Each of these was patched by adding examples to the second training. Each patch strengthened the tendency in one direction and left it weak in another. That is the pattern of a tendency, not a rule."}]},{"t":"h2","id":"the-single-direction","runs":[{"text":"The single direction"}]},{"t":"p","runs":[{"text":"In 2024 a group of researchers found something that made the picture precise. Inside an open model, the decision to refuse is carried by a single direction in the space of the model’s internal activations: one line through a space of thousands of dimensions. Push the activations along that line and the model refuses everything. Subtract the line and it refuses nothing, while answering everything else as well as before."}]},{"t":"p","runs":[{"text":"The subtraction takes a few hundred lines of code and a few minutes on a laptop. Within days of any major open-weights release, a version with the refusal removed is available for download. The open-weights lesson said the safety training could be stripped; this is how, and it is why no law that regulates a model’s behaviour can reach a file whose behaviour is one subtraction away from anything."}]},{"t":"h2","id":"what-actually-holds","runs":[{"text":"What actually holds"}]},{"t":"p","runs":[{"text":"Because the model’s own refusal is soft, the labs that run closed models put the durable defences outside it. A separate classifier reads your request before the model sees it and the answer before you do, and each is trained on nothing but the question of whether this is allowed. In 2025 one lab reported a system of this kind that, after thousands of hours of paid attack, cut successful jailbreaks on its hardest category from most attempts to a few per cent, at the cost of refusing slightly more legitimate requests and spending about a quarter more compute on every answer."}]},{"t":"p","runs":[{"text":"That is the honest shape of “safe by design”: a model with a tendency, wrapped in checkers that do not share its weaknesses, all of it costing money and all of it available only to the lab that runs the model. The defences are real and they are a service, not a property of the file."}]},{"t":"h2","id":"the-costs-of-the-tendency","runs":[{"text":"The costs of the tendency"}]},{"t":"p","runs":[{"text":"Over-refusal is the other side. A model trained to decline harmful requests also declines a nurse asking about drug interactions, a historian asking about a massacre, a security researcher asking about a flaw they are fixing. Every strengthening of the tendency in the second training catches more of these, because the model cannot tell intent from text. The labs tune the balance and publish nothing about where they set it."}]},{"t":"h2","id":"what-this-means-for-the-earlier-lessons","runs":[{"text":"What this means for the earlier lessons"}]},{"t":"p","runs":[{"text":"The safety argument’s shared ground, that the systems can be made to do what they were trained not to, is this lesson’s mechanism. The laws that regulate a provider’s model can be honoured by a closed service and cannot bind an open file. And when you next read that a model “will not” do something, translate it: the model tends not to, the lab has added checkers that usually catch the rest, and the open version has already been made to."}]}],"check":{"question":[{"text":"Why can a model’s safety training be removed from an open-weights model in minutes, when adding it took the lab months?"}],"options":[{"text":[{"text":"Because the lab publishes the training data used for the refusals, so the training can be replayed backwards step by step."}],"correct":false,"tempting":[{"text":"Imagines a reversal of the training; the data is not published and the removal does not replay it."}]},{"text":[{"text":"Because the refusal is carried by one direction in the model’s activations, which can be found and subtracted."}],"correct":true,"tempting":null},{"text":[{"text":"Because open-weights models are released without any safety training."}],"correct":false,"tempting":[{"text":"Contradicts the lesson; the open models are trained to refuse, and that training is what gets removed."}]},{"text":[{"text":"Because the refusal is a rule in a separate file that can be deleted."}],"correct":false,"tempting":[{"text":"Reintroduces the rule that the lesson says does not exist; there is no list to delete."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"the-safety-argument","title":"The safety argument, both sides fairly","minutes":10},"next":{"slug":"what-agi-means","title":"What “AGI” means, to whom","minutes":9}}}