Why a refusal can be trained but not guaranteed

The refusal is not a rule

When a chatbot declines to explain how to make a weapon, most people picture a rule: a list of forbidden topics checked before the answer. There is no list in the model. The second training, from the chatbot module, showed it examples in which a refusal was the preferred answer, and it learned a tendency: given text of this shape, the likeliest good continuation is a polite no.

A tendency can be steered around. Everything in this lesson follows from that.

The jailbreaks, and what each exploits

The earliest trick, from 2022, was to ask the model to play a character who had no rules. It worked because the training had shown few examples of refusing in character, so the tendency was weaker there. Asking for a forbidden thing as a story, a poem or a grandmother’s bedtime recitation worked the same way.

Translating the request into a language with little training data worked because the refusal tendency had been trained mostly in English. A 2023 study found that requests translated into several low-resource languages got through a leading model’s safeguards most of the time.

Filling a long context with hundreds of invented dialogues in which an assistant complied, then asking, worked because the model continues patterns, and the pattern in the window outweighed the tendency from training. That one was published by a lab in 2024 against its own models.

And in 2023 researchers showed that a string of apparently meaningless characters, found by automated search, could be appended to a request and switch off the refusal, and that strings found against open models often transferred to closed ones. Nobody had to understand why.

Each of these was patched by adding examples to the second training. Each patch strengthened the tendency in one direction and left it weak in another. That is the pattern of a tendency, not a rule.

The single direction

In 2024 a group of researchers found something that made the picture precise. Inside an open model, the decision to refuse is carried by a single direction in the space of the model’s internal activations: one line through a space of thousands of dimensions. Push the activations along that line and the model refuses everything. Subtract the line and it refuses nothing, while answering everything else as well as before.

The subtraction takes a few hundred lines of code and a few minutes on a laptop. Within days of any major open-weights release, a version with the refusal removed is available for download. The open-weights lesson said the safety training could be stripped; this is how, and it is why no law that regulates a model’s behaviour can reach a file whose behaviour is one subtraction away from anything.

What actually holds

Because the model’s own refusal is soft, the labs that run closed models put the durable defences outside it. A separate classifier reads your request before the model sees it and the answer before you do, and each is trained on nothing but the question of whether this is allowed. In 2025 one lab reported a system of this kind that, after thousands of hours of paid attack, cut successful jailbreaks on its hardest category from most attempts to a few per cent, at the cost of refusing slightly more legitimate requests and spending about a quarter more compute on every answer.

That is the honest shape of “safe by design”: a model with a tendency, wrapped in checkers that do not share its weaknesses, all of it costing money and all of it available only to the lab that runs the model. The defences are real and they are a service, not a property of the file.

The costs of the tendency

Over-refusal is the other side. A model trained to decline harmful requests also declines a nurse asking about drug interactions, a historian asking about a massacre, a security researcher asking about a flaw they are fixing. Every strengthening of the tendency in the second training catches more of these, because the model cannot tell intent from text. The labs tune the balance and publish nothing about where they set it.

What this means for the earlier lessons

The safety argument’s shared ground, that the systems can be made to do what they were trained not to, is this lesson’s mechanism. The laws that regulate a provider’s model can be honoured by a closed service and cannot bind an open file. And when you next read that a model “will not” do something, translate it: the model tends not to, the lab has added checkers that usually catch the rest, and the open version has already been made to.