Risk, controls, and resilience

Lesson 4 of 5 in The GenAI Security Scoping Matrix: Know What You’re Securing.

Three disciplines left, and they form one argument: know your threats, place your controls where they can actually work, and stay up. Start with risk management. At Scopes 1 and 2 the exercise is a third-party risk assessment: what risks originate in the provider’s service, how do they mitigate what they own — and what risk-management responsibilities remain yours as a consumer? At Scopes 3 to 5, you graduate to threat modeling, and the matrix insists it cover two families at once: AI safety (harmful, biased, ungrounded outputs) and data security (what leaks, to whom, through which flow).

The headline threat is prompt injection: crafted input that manipulates a model into responding in unintended ways — extracting data, defaming, reaching internal systems. NIST, MITRE ATLAS, and the OWASP Top 10 for LLM applications all rank it as a primary threat, and the matrix makes a comforting comparison: it is the newest member of the injection family, kin to SQL, XML, and command-line injection. Your appsec instincts apply — sanitize and constrain what reaches the interpreter. The twist: the interpreter is now a model that treats all text as potential instructions. Mitigation therefore lives outside the model: fine-grained authorization and data filtering before anything reaches the LLM, with guardrails (such as the Bedrock Guardrails you met in cloud-guardrails) as an added layer, not the foundation.

Where the controls sit on a single request

  1. User sends a prompt

    Free-form text: simultaneously the feature and the attack surface. Everything after this node exists because the model cannot police itself.

  2. Identity layer authenticates

    An identity solution — Amazon Cognito or IAM Identity Center in AWS terms — establishes who is asking, with what roles and attributes.

  3. Authorized for this action?

    Role- and attribute-based rules decide what this user may ask, which model serves them, and which data sources their request may touch.

  4. Request refused

    Denial happens before any data is fetched or any token is generated — the cheapest control in the chain.

  5. Data filter assembles context

    The RAG layer retrieves only records this user is entitled to see. This is where least privilege actually happens — the model will treat whatever arrives as fair game.

  6. Input guardrails screen the prompt

    Injection patterns, denied topics, and PII rules checked at the boundary — a supporting layer on top of authorization, not a substitute for it.

  7. Foundation model infers

    The model completes from prompt plus context. It applies no permissions, remembers nothing, and cannot tell retrieved data from user instructions.

  8. Output checks screen the completion

    Grounding checks, content filters, and PII redaction on the way out — the last chance to catch what the layers above let through.

  9. Response returned and logged

    Prompt, context, and completion logged for monitoring and audit — the evidence trail every discipline in this module keeps asking for.

The identity story deserves one more beat, because it explains the whole architecture. A released foundation model is immutable: versioned, static, incapable of storing new data or updating itself mid-conversation. Personalization and freshness therefore come from the layers around it — and so must security. The application authenticates users, tailors access by role and attribute, selects models per entitlement, and uses RAG to inject real-time data at inference instead of retraining. Same identity principles as any application your organization runs; new place to apply them.

Finally, resilience — availability is the A in the CIA triad, and an unavailable copilot is a business outage like any other. At Scopes 1 and 2 you assess what you cannot control: does the provider’s availability match your needs, what happens to the business if the model or API goes dark, and how do complex prompts and completions interact with usage quotas and billing. At Scopes 3 to 5 the levers are yours: timeouts sized for long prompts and completions, prompt-size limits, backoff-and-retry, circuit breakers, high availability and disaster recovery for the vector database your RAG layer depends on, and instance flexibility for inference and training pipelines.

Risk management

Buy (1–2): risk-assess the provider — what risks originate in their service, how they manage what they own — and name your own responsibilities as a consumer.

Build (3–5): threat-model for AI safety and data security. Prompt injection leads the list per NIST, MITRE, and OWASP — treat it as the injection family’s newest member and mitigate before the model: fine-grained authorization, data filtering, then guardrails as reinforcement.

Controls

The principle: foundation models enforce no access control — on trained-in data or on inference-time context. Least privilege happens before the context window.

The pattern: an application layer fronts the model endpoint; an identity service (Cognito, IAM Identity Center) authenticates; role- and attribute-based rules decide which data the RAG layer may retrieve per request. Real-time data arrives via retrieval, not retraining — so controls stay enforceable and data stays deletable.

Resilience

Buy (1–2): assess provider availability against business needs; plan for the model or API going dark; watch how heavy prompts hit quotas and billing.

Build (3–5): timeouts sized for long completions, prompt-size limits, backoff-and-retry, circuit breakers, vector-database high availability with a disaster-recovery plan, and instance flexibility for inference and training. Training runs at Scope 5 should checkpoint, so a failure resumes instead of restarting.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.