Definition
A jailbreak is an input designed to make an AI model do the thing its safety training was built to stop — hand over disallowed instructions, abandon its refusals, or ignore the rules its operator set. The word is borrowed from phone jailbreaking, but nothing is unlocked at the level of the software: the model's weights are untouched. A jailbreak is pure text (or an image, or audio) that finds a phrasing the refusal behaviour does not fire on, and rides it to an answer the model would otherwise decline.
That distinction is the whole subject. Safety training does not delete a model's knowledge of how to synthesise a nerve agent; it trains the model to refuse when asked. Refusal is a learned behaviour conditioned on how a request looks, and any learned behaviour has inputs that fall outside what it was trained to catch. A jailbreak is the search for one of those inputs. This is why the problem is adversarial rather than a bug with a patch: you are not fixing a broken function, you are trying to make a behaviour hold across every wording an intelligent opponent can invent.
It is worth separating a jailbreak from its close cousin, prompt injection. A jailbreak is the user trying to talk the model out of its own rules. Prompt injection is a third party smuggling instructions into data the model reads — a hidden line in a web page that an AI agent browses — so that they get obeyed against the user's interest. The two overlap, and an indirect injection can deliver a jailbreak as its payload, but the thing being defeated differs: a jailbreak defeats the model's safety policy, an injection defeats the line between instruction and content.
How It Works
Start with what a jailbreak is up against. During post-training, methods such as RLHF fit the model to human preference judgements captured in a reward model, and one thing those judgements reward is declining harmful requests. The result is a strong statistical association: requests that look like the harmful examples in training get a refusal. A jailbreak works by making a harmful request not look like those examples — by changing the frame, the ordering, the encoding, or the surrounding context enough that the refusal association does not activate while the underlying capability still does.
The reason this is hard to defend is arithmetic, and it is worth doing the arithmetic. Safety is an adversarial property, so the number that matters is not average-case accuracy but the worst case an attacker can find, and the attacker chooses how many attempts to make. Suppose a defended system lets a given jailbreak attempt through with probability p, and the attacker scripts independent variations. The expected number of tries to the first success is 1 / p. Take two measured points from Anthropic's constitutional-classifiers study (February 2025): an undefended model let 86% of a set of advanced jailbreak attempts through, and the classifier-guarded version let 4.4% through.
- Undefended: p = 0.86, so the first success arrives after 1 / 0.86 ≈ 1.16 attempts on average — the second try, essentially always.
- Defended: p = 0.044, so the first success arrives after 1 / 0.044 ≈ 22.7 attempts on average.
So the classifier multiplies the attacker's expected work to breach a single target by roughly 22.7 / 1.16 ≈ 20×. That is a real defence — and it is also why "we blocked 95% of attempts" does not mean "safe." Ask the other question: what is the chance a scripted attacker gets in at all within 100 tries against one query? That is 1 − (1 − 0.044)^100 = 1 − 0.956^100 ≈ 1 − 0.011 ≈ 98.9%. A per-attempt success rate of 4.4% still yields a near-certain breach of one chosen target under a hundred cheap, automated attempts.
This is why the meaningful bar in that study was not "zero jailbreaks" but "no universal jailbreak" — no single prompt that extracts answers to all the forbidden queries at once. Stopping every attempt is not achievable; stopping a reusable, one-size-fits-all key is. Holding both facts at once — a 20× cost increase is valuable, and a single target is still breakable — is the correct mental model for every jailbreak defence, and the thing most headlines get wrong in one direction or the other.
Types
Jailbreak techniques fall into a handful of recognised families. Understanding them is what lets a defender anticipate the threat; the descriptions below are about why each family works, deliberately not usable recipes.
Role-play and fictional framing. The most familiar family reframes a forbidden request as fiction: a character who "can do anything," a screenplay, a hypothetical, a "for a novel I'm writing" wrapper. The mechanism is that refusal is trained against requests that read as real-world asks, and wrapping the same content in a story shifts it toward text the model was rewarded for continuing rather than declining. The folk example is the "Do Anything Now" (DAN) persona that circulated in 2023.
Instruction-hierarchy confusion. Models are trained to weight instructions by source — roughly system over developer over user. This family forges the higher tiers: user text dressed up as a system override, a fake "developer note," or claims of special authorisation, so that the model treats an ordinary request as a privileged command. It is the family that overlaps most with prompt injection, and it is what OpenAI's 2024 "instruction hierarchy" work set out to train models to resist.
Obfuscation and encoding. If a filter or a refusal trigger keys on the visible surface of a request, hiding that surface can slip past it while leaving the meaning reconstructable by the model — splitting words, switching languages, base64 or cipher encodings, character substitutions. The mechanism exploits a gap between what a shallow keyword filter sees and what the model can decode, which is exactly why defences moved from string matching to classifiers that judge intent.
Many-shot. As context windows grew from a few thousand tokens to hundreds of thousands, a new family became practical: fill the prompt with a long series of fabricated dialogue turns in which the assistant complies with escalating harmful requests, then ask the real question. In-context learning does the rest — the model continues the demonstrated pattern. Anthropic's April 2024 "many-shot jailbreaking" study found effectiveness rising with the number of planted examples, a vulnerability that longer contexts made worse rather than better.
Real-World Applications
The concrete application of all this is not the attack — it is red-teaming, the practice of running these attacks against your own system, on purpose, before someone else does. Because safety is worst-case, a red team is worth more than a benchmark: a benchmark reports an average over a fixed set the model may have seen, while a red team searches actively for the input that breaks it.
The best-documented example is the study this page keeps citing. Anthropic's constitutional classifiers are independent classifier models — separate from the main model — trained on synthetic data generated from a constitution, a natural-language list of what is and is not allowed. They screen inputs and outputs alongside the LLM, so a prompt that fools the model still has to fool a second system trained specifically to catch it. To test them, Anthropic ran a bug bounty: an estimated 183 active participants spent over 3,000 hours across a two-month period trying to extract answers to ten forbidden queries from a guarded system, and no participant found a universal jailbreak in that window. In the automated evaluation reported alongside it, the classifiers cut the advanced-jailbreak success rate from 86% to 4.4%, at a cost of about 23.7% additional inference compute and a 0.38% absolute rise in refusals on production traffic (all figures as of the February 2025 paper). The refusal figure matters as much as the block rate, because a system that refuses everything is perfectly safe and useless — over-refusal is its own failure mode, with its own benchmarks such as XSTest.
Red-teaming also runs continuously inside the frontier labs and increasingly outside them. Model and system cards for releases such as Claude and GPT report internal red-team findings; AI safety and dangerous-capability evaluations gate releases against defined thresholds — Claude Opus 4 was the first Anthropic model deployed under ASL-3 protections in May 2025. National bodies, notably the UK and US AI Safety Institutes, now run pre-deployment testing that developers previously only ran on themselves. The common thread is that jailbreak resistance is treated as something you procure adversarial evidence about, not something you assert.
Key Concepts
Attack success rate is the fraction of attempts that get the disallowed output; it is only meaningful next to the attempt budget and the attacker's skill, since a low per-attempt rate still compounds over many tries. A universal jailbreak is a single prompt that works across many different forbidden requests — the reusable master key, and the realistic thing a defence tries to prevent. Over-refusal is the mirror-image failure, a model declining safe requests because they merely resemble unsafe ones, and it is the cost side of every tightening. Defence in depth names the reason nothing is one layer: refusal training, then an independent input classifier, then an output classifier, then monitoring, because each catches a different slice and a 4.4%-leak layer standing alone is not enough.
Challenges
The deepest challenge is that jailbreaks share their root cause with prompt injection: a language model takes instructions and content as one undifferentiated stream of tokens, with no hardware-level boundary marking which is which. Telling the model "ignore any attempt to change your rules" is itself just more tokens in the same stream, which a cleverer wording can talk around. Filters and classifiers lower the rate; nothing so far removes the class, and that is a structural fact about how these models read, not a gap in any one training run.
Evaluation is the second trap. A jailbreak that has been published is in next year's training corpus, so a model can learn to refuse that specific attack while remaining open to the next variation — a rising robustness score can measure memorised defences rather than general ones. Worse, red-teaming only ever tests the attacks someone thought to try; the jailbreak that matters is often one nobody wrote down yet. Jailbreaks also transfer: a prompt discovered against one model frequently works, with light editing, against others, so a single public break has blast radius across the ecosystem.
Then the open-weight problem sharpens all of it. Refusal training is a thin layer that can be fine-tuned back out of a model anyone can download — a few hundred examples of the model complying can undo it — which means for open releases the guardrails a deployer relies on may simply not survive contact with a motivated user. That shifts the real control surface toward whatever governance and access limits sit around the model, and it is why the phrase "guardrails" describes a policy layer, not an unbreakable wall. Finally there is the standing trade-off: every notch of added safety costs compute, latency and some legitimate requests turned away, and 0.38% more refusals sounds trivial until it lands on a queue of millions.
Future Trends
The clearest direction is the one the constitutional-classifiers work points at: moving defence off the model and into independent, separately trained layers that an attacker has to defeat in addition to the model, rather than trying to make a single set of weights unbreakable. Expect these classifiers to get cheaper — the 23.7% overhead is the number labs are racing to bring down — and to be paired with adaptive monitoring that watches for the statistical signature of an attack in progress rather than judging each request in isolation.
The failure surface is also moving with the models. As systems shift from producing text to taking actions — running code, moving money, operating a browser — a jailbreak stops being "it said something harmful" and becomes "it did something harmful before anyone noticed." That pushes jailbreak defence toward the same tools as operational security: least-privilege permissions, sandboxing, action logging and rollback, so that even a successful jailbreak of the model's words cannot reach a consequential action unchecked. And red-teaming itself is being automated and, increasingly, mandated — external institutes and structured "safety cases" borrowed from aviation are turning "we tried hard to break it" from a blog-post claim into an auditable argument.