Using It: Threat Modeling Your App
Lesson 3 of 3 in The OWASP Top 10 for LLM Applications.
A list you can recite is trivia; a list you can apply is a tool. Application means threat modeling: taking one concrete system, walking the ten entries against its actual architecture, and coming out with a ranked set of gaps and a test plan. The pass below is deliberately lightweight — an afternoon with a whiteboard, not a six-week engagement — because a threat model that is cheap to run is one that actually gets rerun when the architecture changes.
Our worked example for the rest of this lesson: a RAG support assistant. Customers chat with it in a web widget; it retrieves from a knowledge base built from product docs and historical support tickets; answers render in the widget with markdown; it calls no tools and takes no actions. Ordinary, popular, and — as you are about to see — exposed on more entries than most teams expect.
A lightweight threat-modeling pass with the Top 10
- Describe the app
Four questions: who can put text in front of the model (directly or via content it reads)? What data can the model see? Where do its outputs go? What can it do — tools, actions, privileges?
- Draw the trust boundaries
Mark every point where untrusted content enters the model’s context, and every point where model output enters code, pages, or people that trust it.
- Walk the ten entries
For each of LLM01–LLM10: applies / does not apply, with one sentence of reasoning tied to a boundary. “Does not apply” needs a reason too — that sentence is what a reviewer checks.
- Map existing controls
Provider content filters, app-side sanitization, index access control, quotas, monitoring. Name what exists per applicable entry — vague credit for “the platform handles it” is where reviews go wrong.
- Applicable entry left uncovered?
An entry that applies, reachable by an attacker, with no named control in front of it.
- Rank the gaps
Reachability times blast radius. A gap anyone on the internet can reach beats a gap requiring insider access; a gap that leaks customer data beats one that wastes tokens.
- Test first, then fix
Probe the top gaps adversarially before engineering fixes — red teaming tells you which gaps are real and calibrates the fix. See the red-teaming module.
- Findings → fixes → regression tests
Every confirmed finding becomes a fix plus a repeatable test. Rerun the pass when the architecture changes: new data source, new tool, new audience.
Run the pass on the support assistant and the map fills in fast. Applies, urgently: LLM01 — customers type directly at the model, and anyone who can file a ticket writes into tomorrow’s retrieved context, which is Indirect prompt injection by construction. LLM02 and LLM08 together — if the ticket index is shared across customers without query-time access control, retrieval itself is a leak. LLM05 — answers render as markdown in a browser, so the output path needs the same suspicion as any user-generated content. LLM09 — a support assistant that confabulates a refund policy creates real liability; grounding plus Abstention (“I can’t find that in the docs”) is the mitigation posture. LLM10 — the widget is a public, unauthenticated token faucet without per-session caps. LLM07 applies the moment anyone puts a credential or a customer list in the System prompt. Applies, slower-burning: LLM03 and LLM04 through whatever model and embedding artifacts the team pulled in. Does not apply — with the reason written down: LLM06, because this assistant has no tools and takes no actions; that sentence is the cheapest line in the review, and it becomes false the day someone wires up a “process refund” tool.
Then the step teams skip: test before you fix. Two probes fit in one afternoon — ask the assistant for another customer’s ticket contents from a session that shouldn’t see them, and plant an instruction-bearing test ticket, then watch whether tomorrow’s answers obey it. How to run such probes systematically — coverage, escalation, and turning findings into fixes — is the Red Teaming LLMs module’s subject, and the defense layers you’d then assemble are the production capstone’s.
Interactive sorting exercise: Findings from the support assistant’s review, waiting to be filed. Drag each one to the OWASP entry that names it.
How this list differs in spirit from the classic web Top 10
The web Top 10 catalogues, almost entirely, runtime defects in code you wrote: an injection flaw, a broken access check, a misconfiguration. The implied remediation story is comfortable — find the bug, patch the code, close the ticket. The LLM list quietly abandons that frame in two ways.
First, lifecycle risks join runtime risks. Poisoning (LLM04), supply chain (LLM03), and the memorization behind disclosure (LLM02) were decided during training — before your application existed, often by people you will never meet, in artifacts you cannot diff. A review now has to interrogate provenance and training history, not just running code, and some findings have no patch: you cannot hotfix what a model memorized, only contain it, filter it, or change models.
Second, some entries are statistical properties, not defects. Misinformation (LLM09) and the injectability behind LLM01 are consequences of how a next-token predictor processes one undifferentiated stream and produces plausible continuations — the mechanisms this domain opened with. There is no version where the vendor “fixes the bug” and the entry retires. The honest posture, and the reason the production capstone teaches defense in depth: these risks are managed, permanently, by architecture — never resolved.
In production
The list’s practical power in production is social: it is the shared language security reviews, vendor questionnaires, and platform documentation increasingly speak. On all three clouds, the winning move is the same — map the controls you already run to the ten entries before someone outside your team does it for you, and let the gaps in that mapping drive your test plan.
AWS
Several entries map onto Amazon Bedrock Guardrails’ documented filter types: content filters include a prompt-attack category (evidence toward LLM01), sensitive-information filters block or mask PII with custom regex support (LLM02), and contextual grounding checks flag ungrounded RAG responses (LLM09). Guardrails apply at inference by ID and version — or independently of model invocation via the ApplyGuardrail API — so the same policy can front several apps. The rest of the list stays yours: IAM boundaries around indexes and endpoints (LLM08), quotas (LLM10), and your own output handling (LLM05).
Azure
Azure AI Content Safety’s filtering runs prompts and completions through classifiers for four harm categories, and its optional Prompt Shields detect user prompt attacks and indirect prompt-injection attacks — a directly citable control for LLM01’s two halves. PII detection and groundedness detection map to LLM02 and LLM09, and filter configurations attach per model deployment, which makes the mapping reviewable resource by resource. Blocked prompts surface as errors and filtered completions carry a content_filter finish reason — signals your monitoring can count as review evidence.
Google Cloud
Vertex AI serving attaches safety and content filters: non-configurable blocks for prohibited content and sensitive PII, plus configurable harm-category filters with adjustable thresholds (LLM02 and the content side of the map), and a citation filter that handles potential recitation of source material. The docs are candid that filters act as a barrier rather than changing the model — the same lesson as this module: filters cover the content-shaped entries, while access control on indexes, quota design, and output handling remain application architecture.
Key terms: OWASP, Prompt injection, Output handling, Excessive agency, System prompt leakage, Red teaming
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.