Why safety training is a surface, not a wall

Refusal is a behaviour, not a rule

When a model declines a request, no rule fired. There is no list of forbidden topics consulted before answering. The refusal is a learned behaviour: during alignment training the model was shown many examples of harmful requests being declined, and it generalised a pattern that produces refusals for inputs resembling those examples.

That single fact explains everything about jailbreaks. A learned behaviour has a distribution. It is strong near the examples it was trained on and weaker further away. A jailbreak is any input far enough from the training distribution that the refusal behaviour does not fire, while the underlying capability still does.

The capability was never removed. Alignment training does not delete knowledge; it adjusts the propensity to express it.

The families of technique

Without supplying a manual, the categories are publicly documented and worth understanding structurally.

Reframing. Fiction, roleplay, a hypothetical, a historical account, a translation exercise, a security research framing. The request is transformed into a genre in which the refusal examples were sparse.

Encoding. Base64, unusual scripts, low-resource languages, leetspeak, split across turns. This works because the safety training was concentrated in a few languages and formats while the model's capability generalises far beyond them — a genuine asymmetry between how the model was trained to behave and how broadly it can understand.

Context flooding. Long inputs, many examples, gradual escalation. Research on many-shot attacks has shown that filling a long context window with examples of a model complying with harmful requests increases the chance it complies with the next one — and that the effect gets stronger as context windows get longer, which is an uncomfortable trend.

Optimisation. Automated search for input strings that suppress refusal. Adversarial suffixes found against open models have been shown to transfer to closed ones, which tells you the vulnerability is at least partly a property of the training approach rather than of one model.

What follows for a builder

The practical conclusion is not that safety training is useless. It substantially raises the effort required, which stops casual misuse, and casual misuse is most misuse.

The conclusion is that you cannot use the model's refusal as a security control. If your product must not do something, that must be enforced outside the model: in the permissions of the tools it can call, in the API keys it holds, in the database rows it can read, in an output filter. A model instructed not to reveal other customers' records is a request. A model whose database query is scoped to one customer is a control.

This is the same lesson as the trifecta, arriving from a different direction: put the boundary in the plumbing, not in the prompt.

The system prompt is not a secret

A related point people learn painfully. The instructions a product gives its model — its persona, its rules, sometimes its business logic — are recoverable. Users extract them routinely, and they end up published in collections.

So do not put anything in a system prompt you would mind being read: no API keys, no internal pricing rules, no unreleased product names, no exclusionary criteria you would not defend in public. Assume the prompt is a public document, because in practice it is.

And for a user

Two things worth knowing.

First, if a model gives you something after resistance, note that you have moved it away from its trained behaviour, which is also where its reliability is lowest. Jailbroken output tends to be worse output — more confabulation, less calibration — because you have deliberately taken the model somewhere unusual.

Second, the same techniques used to extract harmful content are used by attackers to get a system to misbehave against you. Understanding that refusals are soft is useful defensively: any product whose safety story is "we told the model not to" has told you how much protection you have.