Preferences and controls
Lesson 1 of 5 in Tool Scoping and Least Privilege: Making the Dangerous Thing Impossible.
You arrive here from the prompt injection module holding one uncomfortable conclusion: any text the model reads can become an instruction it follows, and no amount of prompt engineering closes that hole reliably. Simon Willison, who named the attack class in 2022, still frames a parameterized-query-style fix — cleanly separating instructions from data — as “extremely difficult, if not impossible” with current architectures.
So stop trying to win that argument. If you cannot guarantee what the model will ask for, engineer what the runtime will allow. Everything in this module is one move: take the dangerous capability away from the loop so that a fully compromised model, doing exactly what an attacker wants, still cannot cause the outcome you fear.
The test for whether a rule is a control is mechanical: could a maximally cooperative attacker inside the context window break it? If the answer is “only if the model chooses to disobey,” it is a preference. If the answer is “no, the call would be rejected before it ran,” it is a control.
Preferences are not worthless — they keep the agent on the happy path, reduce bounced calls, and make behaviour legible. They just belong in a different column of your design doc, one that carries no guarantees. Read the crosswalk below and note the pattern: the control version is almost always less code than the preference version, and it lives somewhere the model has no vote.
| The rule | As a preference (prompt) | As a control (runtime) | Enforcement point |
|---|---|---|---|
Never refund more than $100 | “Refunds up to $100 may be issued directly. Anything above requires approval.” | The | Tool handler / schema validation |
Only email the person who opened the ticket | “Send email only to the ticket requester. Never CC anyone.” | The email tool takes no | Tool signature — the parameter does not exist |
Only touch this one repository | “You are working on | The session gets a token scoped to one repo. Other repos return 404 to the agent, exactly as they would to a stranger. | Scoped credential issued per task |
Never send customer data to an outside domain | “Never include order or payment data in outbound requests to third parties.” | The sandbox has no route to the internet except an allowlisted proxy; everything else fails at the network layer, logged. | Egress proxy / network policy |
Key terms: least privilege, permission surface, egress control, scoped credentials, blast radius, prompt injection
Objection: “The next model will be robust to injection.”
Maybe partly — and it will not change your design. Injection resistance is a rate, subject to the same non-determinism as everything else the model does, and security controls are judged on the worst case, not the average one. A model that resists 99.9% of injected instructions still hands an attacker one success per thousand hostile documents, and attackers get to send a thousand documents. Robustness improvements make your preferences better preferences; they do not turn them into controls.
Objection: “We bought a guardrail classifier that catches 95% of attacks.”
Willison’s objection to detection-first defences is worth memorising: in security, catching ~95% of attacks is a failing grade, because a motivated adversary iterates until they find the 5%. Guardrail classifiers earn their place as detection — they tell you you are under attack, they raise the cost of a casual attempt — but a control you can bypass by rephrasing is not a containment boundary. Layer them on top of scoping; never in place of it.
Objection: “We validate the model’s output before acting on it.”
Good — as long as the validator is checking the call, not the story. Validating “does this tool call satisfy my invariants” is a control. Validating “does the model’s explanation of why it wants this seem reasonable” is a preference with extra steps, because the explanation is generated by the same compromised process that chose the call. Never let the model’s narration be an input to the authorization decision.
Objection: “Locking everything down will make the agent useless.”
This is the only serious objection, and the answer is scope, don’t amputate. The Fernway support agent in the foundations walkthrough still issues refunds — it just cannot issue one above $100, to an order it never looked up, or after its tenth tool call. Capability minus the tail of the distribution you were never willing to accept anyway. When scoping genuinely removes needed function, that is the signal to split the task into two sessions with different permissions rather than to widen one.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.