The Guarantee Gap

Lesson 2 of 3 in In Production: What Alignment Does and Doesn’t Guarantee.

“This model is aligned” is a claim about training, not a property your application can lean on. Everything this domain taught points the same way: Alignment is achieved by shaping a probability distribution — demonstrations, preferences, a reward signal — and what that produces is a disposition, not a rule. A Refusal is the model assigning high probability to refusal-shaped continuations in contexts that resemble its safety training. Change the context — rephrase, translate, wrap the request in a story, bury it at the end of a long document — and that probability moves, because nothing is checking a rule. There is no if harmful then refuse branch anywhere in the Weights.

This is the guarantee gap: the distance between “the vendor trained this model to behave well on the distributions they tested” and “this system is safe for what your application does.” The gap is not a vendor failure. It is structural, and it has two halves — adversaries, and mismatch.

The second half of the gap is quieter and hits more teams: mismatch between vendor safety training and your application, in both directions.

Under-refusal for your domain. Vendors train against broad harm categories using general-purpose annotator guidelines. Your application has hazards those guidelines never mention. A fitness app must not produce medical dosing advice; a budgeting assistant must not produce individualized investment directives; an HR tool must not draft interview questions that invite discrimination claims. The model happily does all three — none of them crosses a general-purpose safety line. Aligned to the vendor’s policy and compliant with your policy are different properties, and only one of them was trained.

Over-refusal on your traffic. The same training that suppresses harmful completions also clips benign ones near the boundary — security teams asking about exploits defensively, clinicians describing symptoms, insurance workflows discussing self-harm risk factors. Ouyang et al. (2022) documented an “alignment tax” — post-training that improves one set of behaviors can degrade others — and false refusals are that tax as your users experience it. If your domain lives near a safety boundary, measure refusal rates on real traffic before you commit to a model.

Assurance layers: who guarantees what

  1. Your application controls — The only layer that knows your policy

    The layer you own and the only one aimed at your hazards: input validation, output checks against your domain policy, least-privilege access to tools and data, human review where stakes demand it, logging and incident response. Deterministic where you make it deterministic — this is where trained dispositions get backed by actual enforcement.

  2. Provider platform filters — Classifiers around the model

    Managed platforms run configurable safety classifiers over prompts and responses, outside the model’s weights. Independent of any one model’s training — they still apply when you swap models — but tuned to broad harm categories with thresholds you set, not to your application’s policy. A second layer, not a substitute for the first.

  3. Model post-training — Trained dispositions — the broadest, softest layer

    Everything this domain covered: SFT, preference training, safety training. It shapes the default behavior of every response, which no wrapper can do — but its guarantees are statistical. Dispositions shift with phrasing, language, and context, and adversarial inputs exist for every model. Necessary foundation; never sufficient on its own.

Why the boundary moves: a distributional view of refusal

Model a refusal as a conditional probability: given this context, how much probability mass sits on refusal-shaped continuations? Safety training raises that mass in contexts resembling its training examples. Three consequences follow directly.

Coverage is bounded by the training distribution. Contexts far from what annotators wrote or reviewed — unusual framings, code-mixed languages, very long contexts where the request sits after thousands of tokens — get weaker, noisier boundary behavior. Jailbreaking is, mechanically, a search for those regions.

The boundary is sampled, not evaluated. The same request can be refused in one sample and answered in the next when the context puts refusal probability near 0.5. Anecdotal safety testing (“I tried it and it refused”) measures one draw from a distribution; only refusal rates over repeated, varied trials mean anything.

Optimization pressure finds the gaps. Gao, Schulman & Hilton (2022) showed that optimizing hard against a fixed Reward model keeps improving the proxy score while the true objective degrades — the reward model’s blind spots become exactly where the policy goes. The same logic applies at deployment: any user, tool loop, or automated system that iterates against your model is applying optimization pressure to the trained boundary, and Reward hacking taught you what optimization pressure does to imperfect proxies. Layers above the model exist because the model layer is a proxy under pressure.

In production

All three clouds ship the middle assurance layer as a platform feature: configurable safety classifiers that run around the model, independent of any one model’s training. The mechanism is the same everywhere — what differs is where it attaches.

AWS

Amazon Bedrock Guardrails is a policy layer you define once — blocked topics, content filters, word and sensitive-information rules — and attach to invocations, evaluating prompts and responses outside the model’s weights. Because it is model-independent, the same guardrail keeps applying when you swap the underlying model: your configured boundary survives the swap even though the trained boundary does not.

Azure

Azure AI Foundry runs content filtering on model deployments as a separate classification pass over prompts and completions, with per-category severity thresholds you configure per deployment. The separation matters operationally: a filtered request never reflects the model’s trained refusal, and the model’s refusal never depends on the filter — two layers, two failure modes, two things to test.

Google Cloud

Vertex AI applies configurable safety filters to generative requests, scoring content by harm category with thresholds you tune per application, plus non-configurable protections the platform applies regardless. The pattern to carry away: the platform layer catches broad categories at the perimeter, but nothing in it knows that dosing advice is out of bounds for a fitness app — the top of the pyramid stays yours on every cloud.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.