Why Refusals Break
Lesson 4 of 4 in Prompt Injection and Jailbreaks: The Model-Level View.
Everything in this module keeps landing on the same floor, and this lesson names it. A Refusal is not a rule. There is no access-control list in the model, no policy engine evaluating requests, no code path that returns DENIED. Refusal behavior is installed by Reinforcement learning from human feedback (RLHF) and its preference-training relatives: raters preferred responses that declined certain requests, the reward signal made declining high-probability in those regions of input space, and the result is a disposition — a statistical tendency, encoded in continuous weights, strongest on inputs resembling the training data that shaped it. What that training does and does not guarantee is a module of its own — In Production: What Alignment Does and Doesn’t Guarantee — but the security consequence belongs here: a disposition has a domain of validity, and everything outside that domain is attack surface.
This reframes lesson three entirely. The jailbreak families are not five clever tricks; they are five ways of performing distribution shift — moving an input from the region where refusal is near-certain toward regions where the training was thin. In a continuous space there is no fence between those regions. There are only gradients of reliability, and the attacker, not the defender, chooses where on that surface each request lands.
Can training close the gaps? The published trajectory says: it can move them and shrink them, not eliminate them. Wallace et al. (2024) trained models on an explicit Instruction hierarchy — teaching them to privilege system instructions over user instructions over third-party content like retrieved documents — and reported improved robustness to injection and extraction attacks. Read the mechanism honestly, though: this is more training. It reshapes the disposition, raising the reliability of the privilege ordering on inputs like those it saw; it does not convert the ordering into a checked rule. The paper’s own framing is training models to prioritize, and prioritization learned by gradient descent inherits the same domain-of-validity caveat as the refusals it protects.
That is why the field describes an arms race, empirically observed and not solved. Providers patch the disposition against known attack styles; attackers shift distribution again; GCG showed the shifting can itself be automated. Published defenses report measured reductions on specific attack suites — a real and valuable thing — and none, to date, claims the class is closed. For a defender the posture follows: track the empirical literature, assume today’s refusal reliability is a snapshot, and never let a safety property your system depends on live only in the model’s weights. Sort the following claims with that lens.
Interactive sorting exercise: Where does each safety property actually live? Sort each claim: a trained disposition (in the model’s weights — probabilistic, erodible by prompting) or an enforceable control (outside the model — holds no matter what the model generates).
In production
In production, the enforceable layers are the ones that run regardless of what the model generates: the provider’s safety filters wrapped around the model, and your own input and output handling in application code. The model’s refusals are a valuable statistical layer — but on every cloud, the honest architecture treats them as exactly that, never as the control. (One caveat travels with all three columns: provider filters are classifiers, so they are probabilistic layers you can enforce the presence of — not oracles.)
AWS
Amazon Bedrock Guardrails evaluates user inputs and model responses as a layer separate from the model itself — documented filter types include content filters (whose predefined categories include Prompt Attack), denied topics, sensitive-information filters for PII, and contextual grounding checks. Two properties matter architecturally: guardrails are versioned and attached at inference by ID, so the safety configuration is your artifact rather than the model’s mood, and the ApplyGuardrail API runs checks independently of model invocation — letting you evaluate text on your own input and output paths, which is where enforcement belongs.
Azure
Azure AI Foundry runs prompts and completions through a built-in content filtering system powered by Azure AI Content Safety, with optional Prompt Shields targeting user prompt attacks and indirect prompt-injection in documents. The enforcement contract surfaces in your code: blocked prompts return an HTTP 400, and filtered completions arrive flagged with a content_filter finish_reason — signals that exist whether or not the model would have complied. Your application must actually branch on them; a finish_reason nobody checks is a control nobody has.
Google Cloud
Vertex AI attaches safety and content filters to model serving: non-configurable filters for prohibited content, plus configurable filters with per-category blocking thresholds, and the API reports why content was blocked via explicit block and finish reasons. Google’s own docs draw this module’s distinction — filters act as a barrier rather than changing the model, while system instructions steer the model’s behavior itself. Steering is the disposition; the barrier plus your own input/output handling are the layers you can actually rely on being present.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.