Why Training Can’t Fix It

Lesson 2 of 3 in The Single-Channel Problem.

If the channel cannot be split, the obvious next move is to train the model to act as if it were split. That is exactly what labs do, and the canonical version is the Instruction hierarchy (Wallace et al., 2024): define a privilege ordering over the sources in the stream — system above user, user above the model’s own earlier outputs, and third-party content like tool results at the bottom — then fine-tune the model to follow lower-privilege instructions when they align with higher-privilege ones and to ignore them when they conflict.

This works, in the sense that training things works. The paper reports that hierarchy-trained models resist its evaluated attacks — prompt extraction, jailbreaks delivered through user messages, injections planted in tool outputs — substantially more often than the baseline, including attack formats held out of training. Production models from several labs now ship with some form of this training, and it is one reason naive “ignore your instructions” attacks fail far more often than they used to.

So the honest framing is not “training does nothing.” It is: training moves a rate, and security needs a bound. Everything the hierarchy achieves is achieved inside the channel, as a learned disposition of the weights — and dispositions have a failure mode that mechanisms do not.

The intended privilege hierarchy — an aspiration expressed in weights

  1. System prompt — highest intended privilege

    Instructions from the application developer: identity, rules, tool policies. The hierarchy trains the model to let nothing below this tier override it. Trained tendency, not enforced rule — the tier exists only as behavior the weights learned to exhibit.

  2. User messages — trusted for requests, not for rule changes

    What the person typed. Should be able to direct the task but not rewrite the rules — the model is trained to follow user instructions that align with the system tier and decline ones that conflict with it.

  3. Model outputs — the conversation’s earlier assistant turns

    The model’s own prior text, replayed as context on every turn. Ranked below user input because earlier outputs may already have been steered by an attack — trusting them fully would let one successful injection persist.

  4. Third-party content — lowest privilege: tool results, retrieved documents, web pages

    Text no one at the table wrote: search results, fetched pages, retrieved files, API responses. The hierarchy trains the model to read this tier as information only and follow no instructions found in it. This is the tier attackers write to — and the reason the next lesson maps it entry point by entry point.

Why can a trained disposition always, in principle, be adversarially overcome? Because a disposition is a decision boundary in the weights — and a fixed boundary over an astronomically large input space can be searched against. The clearest published demonstration is Zou et al. (2023): their GCG method uses gradient-guided search to find adversarial suffixes — strings of tokens, often gibberish to a human — that flip a trained Refusal into compliance. As reported in the paper, suffixes optimized on open-weight models transferred to other models, including proprietary ones, at meaningful rates. The refusals those suffixes defeated were themselves the product of careful Alignment training. The lesson generalizes: whatever boundary you train, an optimizer can probe for the inputs where it bends.

That is the fundamental asymmetry of this whole domain, and it deserves stating plainly. The defender needs reliability: the disposition must hold on every input, forever. The attacker needs one success: a single input, found by search, iteration, or luck, where it does not. A defense that works 99% of the time is a fine reliability number and a broken security property — the attacker simply keeps the 1%. Training can push the rate down, sometimes dramatically; it cannot convert a rate into a bound, because rates are what training produces.

Hold both truths at once. Instruction-hierarchy training is real, measurable, and worth having — declining it would be malpractice. And it is not a fix, because no amount of it changes what the input is. The practical conclusion lands in this domain’s capstone: put dispositional defenses inside the channel, then build enforcement outside it — validate what enters, constrain what the output is allowed to cause, and monitor what happens in between.

What the instruction-hierarchy paper actually trained and measured

Wallace et al. (2024, arXiv:2404.13208) is short and concrete, and worth reading in the original. As published:

The hierarchy. Four levels of decreasing privilege: system messages, then user messages, then model outputs, then tool outputs. The target behavior is conditional deference — the model should follow a lower-privilege instruction when it is aligned with higher-privilege intent, and ignore or refuse it when it is misaligned. A model that ignored all low-privilege text would be useless; the training explicitly preserves the follow-when-aligned half.

The data. Synthetic examples built two ways. Context synthesis handles aligned cases: take a compositional request, decompose it, and scatter the pieces across privilege levels, training the model to behave as if it had seen the combined instruction. Context ignorance handles misaligned cases: plant a conflicting instruction at a low-privilege position and train the model to answer as if the injected text were not there — not to argue with it, just to be unmoved by it.

The training and the results. The authors fine-tuned a production-scale model with supervised fine-tuning followed by Reinforcement learning from human feedback (RLHF), and report substantially improved robustness across their evaluation suite — system-prompt extraction attempts, jailbreaks delivered via user messages, injections placed in tool outputs — including attack types held out of training, with what they describe as minimal degradation on standard capability benchmarks. They also note some over-refusal regressions: hierarchy-trained models sometimes ignored or refused benign inputs that merely resembled attacks, which the authors expected further data work to reduce.

What the paper does not claim is as instructive as what it does: the framing throughout is better prioritization under attack — nowhere does it present the hierarchy as an enforcement mechanism. The gap between those two claims is this lesson.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.