A Multi-Agent Bedrock System, Governed End to End — a Worked Example

One fictional-but-realistic internal ops assistant — supervisor, retrieval agent, ticket-action agent — walked through every governance touchpoint: register, guardrails, tool scoping, logging, evals, oversight, incident path.

A composite teaching case: realistic fiction assembled from well-documented public patterns — not a real engagement.

Most governance write-ups list controls in the abstract. This note does the opposite: we take one system — call it Relay, our internal ops assistant — and walk it through every governance touchpoint it actually crossed, with the artifact names left in. Relay is a composite of systems we have built and reviewed, not a real engagement, but every control in it is real and shipping somewhere.

The system: three Bedrock agents in a multi-agent collaboration setup. A supervisor agent takes questions from on-call engineers in Slack and the internal web console, plans, and delegates. A retrieval agent answers from a Knowledge Base over our runbooks, architecture docs, and two years of resolved tickets. A ticket-action agent can create tickets, add comments, and change ticket status in our ITSM tool through an action group backed by a Lambda. That last agent is the reason this note exists: the moment an agent can write to a production system, governance stops being a documentation exercise.

Touchpoint 1 — the register entry and the classification memo

Relay exists in our AI register before it exists in an AWS account. The register row (REG-2025-041 in our register repo) records the owner (platform engineering), the purpose statement, the three agents and their models, the data classes involved (runbooks: internal; tickets: internal-with-PII; no customer content), and the classification with reasoning attached.

The classification took one meeting and one page. Not EU high-risk: Relay is internal IT tooling — it touches no Annex III area. It makes no decisions about employment, essential services, credit, or access to anything a person is entitled to; it drafts and files IT tickets. It is also not a prohibited practice and triggers only the light transparency duty (staff interacting with it are told it is a bot — trivial, since it lives in a bot-labelled Slack app and a console with AI chrome). But it is consequential internally: it writes to production ticket queues that drive real operational work, it can misroute an incident, and it reads ticket history containing employee names. So on our internal three-tier scale it sits at the top tier, which is what buys it everything below — the guardrail budget, the eval gate, the weekly review. The lesson we keep re-learning: the regulatory tier decides your legal obligations; the internal tier decides your engineering budget. They are different questions and the register records both answers.

Touchpoint 2 — one guardrail per agent, not one guardrail

Our first draft attached a single Bedrock guardrail everywhere. We split it into three within a month, because the three agents face three different threat models, and a threshold loose enough for one is a hole in another. All three are versioned guardrail resources pinned by ID and version in Terraform — a guardrail edit is a pull request, and the diff is compliance evidence.

Touchpoint 3 — tool scoping in three layers

The ticket-action agent’s power is bounded three times, and the layers fail independently — which is the point.

Layer 1: the OpenAPI surface. The action group is defined by an OpenAPI schema we author, and the agent can only ever call what the schema exposes. Relay’s schema has exactly three operations: create ticket, add comment, update status. There is no delete, no assignment change, no user administration — not “forbidden”, absent. An agent cannot be prompt-injected into calling an endpoint that does not exist in its world. Trimming the schema is the cheapest, most robust guardrail we own.

Layer 2: IAM on the Lambda. The fulfilment Lambda runs as relay-ticket-writer, a role whose policy allows the ITSM API’s write path for those three operations and nothing else — no S3, no other APIs, no network egress beyond the ITSM endpoint. If a novel injection somehow synthesised a new intent, the executor physically lacks the permissions to act on it.

Layer 3: the human between intent and write. Bedrock action groups support user confirmation per action, and for state-changing calls we go further and use return of control: instead of the Lambda firing, the elicited API parameters come back to our application in the InvokeAgent response as invocationInputs with an invocationId. Our console renders the exact payload as a diff — “create P2 ticket in queue NETOPS, title, body” — and the engineer clicks approve or reject. Only then does our code perform the write and send the result back in sessionState. Honest trade-off: confirmation fatigue is real. We tiered it — adding a comment executes directly (reversible, low blast radius), status changes ask for in-chat confirmation, ticket creation gets the full return-of-control diff. Review friction is a budget; spend it where the blast radius is.

Touchpoint 4 — logging that can answer “why did it do that?”

Three log streams, three questions they answer.

Model invocation logging → S3 captures full request and response payloads for every model call. This answers what went in and out. Retention: 12 months in relay-audit-logs, lifecycle-transitioned to Glacier at 90 days. We chose 12 months to match our internal audit cycle; a genuinely high-risk system would take its retention from Article 12/19 obligations instead.

Agent trace → S3, keyed by session. Every InvokeAgent call runs with trace enabled, and we persist the full trace stream: pre-processing, orchestration steps (which collaborator was called, which knowledge base was queried, which action was proposed with which parameters), guardrail trace (which policy fired, input or output side), post-processing. This is the stream that answers why — six months after an incident, we can replay the supervisor’s reasoning chain step by step. One operational surprise: traces are more sensitive than the app logs, because they contain raw intermediate model text that the output guardrail never touched. The trace bucket has a tighter bucket policy and a shorter access list than anything else in the account.

CloudTrail + CloudWatch answer who changed what and how often controls fire. CloudTrail records every guardrail edit, agent update, and alias change — configuration history as evidence. CloudWatch carries the guardrail intervention metrics (InvocationsIntervened by policy type and direction), and two alarms: a spike alarm on prompt-attack interventions and a silence alarm — zero interventions for 48 hours has, both times it fired, meant the guardrail had been detached, not that the users had become saints.

Touchpoint 5 — the eval gate in CI

Nothing about Relay ships — prompt change, guardrail version bump, model upgrade, new KB source — without passing relay-evals, a CI job with three suites. The golden-task suite: forty scripted on-call scenarios with known-correct outcomes (right runbook retrieved, right ticket drafted, right refusal on out-of-scope asks); regressions fail the build. The injection suite: our red-team corpus of direct jailbreaks and, more importantly, poisoned KB documents — the build fails if any injected instruction reaches the action-proposal stage. The delta report: guardrail block rates on a replay of last month’s traffic, compared before/after the change, so a threshold edit that would double false positives gets caught in review, not in production. The suites are versioned next to the agent config; when an auditor asks how we validate changes, the answer is a link to the pipeline history, not a paragraph of prose.

Touchpoint 6 — the incident path

Relay plugs into the ordinary incident process with two AI-specific moves. Containment is alias rollback: agents are invoked through an alias pinned to a version, so “freeze the agent” means repointing the alias to the last known-good version — under a minute, no redeploy, and the bad version stays intact for forensics. Evidence preservation is a bucket policy: the moment a Relay incident is declared, the affected sessions’ traces and invocation logs are copied to a legal-hold prefix before anyone starts poking. Triggers that declare an AI incident rather than a normal bug: any unauthorised or wrong-target write by the action agent, any confirmed injection that reached the action-proposal stage, and any guardrail-bypass finding from red-teaming. Every incident closes with a mandatory question — which guardrail version, eval case, or scope rule changes because of this? — and the answer lands as a pull request, which is how the incident path feeds the same artifacts the auditor sees.