Threat-modelling a new agent design
Lesson 5 of 5 in The Agent Threat Model.
Threat-modelling an agent is not a document exercise; it is five questions asked in a fixed order, early enough that the answers can still change the design. Ask them at the whiteboard stage and they cost twenty minutes. Ask them after launch and each answer is a migration.
The order matters, because each question narrows the next. Walk a real design through it now — ideally one you are actually building.
Threat-model a new agent design
Interactive decision tree — outcomes:
- One leg — but keep the inventory alive
A closed-input agent with private data is a defensible design and the strongest of the three cuts. Its characteristic failure is organisational, not technical: someone ships "let it read customer feedback" in six months and nobody re-runs the count. Write the input inventory into the design doc, and make adding an input a review trigger.
- Two legs — abuse risk, not exfiltration risk
Untrusted content plus a channel, with nothing confidential in reach: an attacker can make the agent emit content or fire requests outward, so your residual risks are abuse, spam and reputational damage. Rate-limit the outbound tools and log them. Guard the boundary jealously — the day someone mounts an API key "just for the internal wiki", you are at three legs.
- Two legs — confidentiality contained, integrity still exposed
With no exfiltration channel, injection buys the attacker your agent’s reasoning: wrong answers, poisoned memory that persists into later runs, destructive-adjacent decisions. Validate memory writes, verify outcomes out-of-band, and audit relentlessly for accidental channels — rendered images, clickable links, logs a third party can read.
- Split the sessions — the best available answer
Context separation is the containment that survives the fact that you cannot fix the token stream: process untrusted content in a low-privilege session that produces only structured, validated output, then act on it in a privileged session that never sees the raw text. This is Meta’s Rule of Two applied literally — at most two properties per session — and it usually costs less than teams expect.
- Full trifecta with a real gate — acceptable, and now an operational problem
You have chosen supervision over autonomy, which is the sanctioned answer when all three properties are unavoidable. It only stays true if the gate stays real: monitor review latency and rejection rate as live metrics, batch and rate-limit what reaches the reviewer, and keep the log good enough to reconstruct any approved action. Assume the gate degrades unless something measures it.
- Approval theatre — you have the cost of oversight without the protection
A reviewer who sees prose at volume and approves everything is a rubber stamp that also gives you false confidence. This is the human-agent trust exploitation threat, arrived at by neglect rather than by attack. Fix the reviewability first — show the destination, the amount, the diff; cut the volume reaching humans by an order of magnitude — or cut a leg instead.
- Redesign — no chain-closing choke point exists
All three legs, unattended, with many tools any of which could leak: there is nowhere to put a control, so no control will hold. Do not ship it and plan to monitor. Narrow the data leg to the task (per-task credentials, per-session scope), collapse the outbound tools to one auditable path, or split the sessions. Then walk this tree again.
The tree gets you to a verdict. What you hand to a reviewer — or to your future self, six months into feature creep — is five short lists. Write them in the design doc, not in a security tool nobody opens.
| Artefact | What it contains | What it closes | How it decays |
|---|---|---|---|
Input inventory | Every source of text the agent reads, marked trusted or attacker-writable — including tool results, memory and other agents. | Leg 2 blindness: the injection surface you did not know you had. | A new integration lands and nobody adds a row. Make it a merge-request checklist item. |
Data scope | The credentials the agent actually runs with, and what they can reach — not what the task needs, what the token permits. | Privilege compromise and identity abuse: the difference between one folder and an entire estate. | Scopes get widened during an incident and never narrowed afterwards. Re-audit the token, not the intention. |
Egress inventory | Every path by which model-chosen bytes can leave: tools, renderers, links, logs, stores, DNS. | Leg 3, the leg most cheaply cut and most often mis-counted. | A UI change re-enables image or link rendering. Test the channel, do not read the config. |
Irreversible-action list | Actions that cannot be cheaply undone: sends, payments, deletes, deploys, public posts. | Tool misuse, and destructive non-deterministic failure — the Replit-style incident with no attacker at all. | A new tool is added as "read-only" and later grows a write method. Review tools by capability, not by name. |
Telemetry and stop plan | What the trace captures, who watches it, the budgets the runtime enforces, and how the whole thing gets stopped. | Repudiation and untraceability — the threat that turns a one-hour incident into a three-week one. | Traces get sampled down for cost. Keep security-relevant fields at 100%, and test the kill switch on a schedule. |
Tool: Tool Permission Lab — Shrink leg 1 by hand: the Permission Lab lets you scope an agent’s credentials per task and watch which attack chains stop working.
Interactive flashcard deck.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.