Running a review
Lesson 5 of 5 in The Well-Architected Agentic AI Lens: 41 Questions Worth Borrowing.
The lens itself says not every team needs every best practice, and then does something more useful than most framework documents: it names four reading paths, each with a specific short list of questions. Three or five questions, not forty-one. That is the difference between a document people use and a document people cite.
Start from the path that matches where you actually are. If you are between two, take the earlier one — the foundational questions are the ones that get expensive to answer late, because an agent shipped without a permission boundary has to be rebuilt rather than patched.
Building your first agent
AGENTOPS01 · AGENTREL02 · AGENTSEC03 · AGENTSEC08
Scope and success criteria; atomic tasks with least-privilege permissions and clear instructions; agent identity and authentication; input validation and output filtering before you go to production.
Why these four. They are the ones you regret skipping. Everything else can be added later — traces, evals, cost attribution, cost ceilings all bolt on to a running system. A permission boundary does not bolt on: if the agent shipped with a broad service identity and the whole design assumes it, narrowing it later means rebuilding the integration layer. Same for scope. An agent with no written definition of done accretes responsibilities until nobody can say what it is for, and by then the answer to “what should this agent not do?” is nobody’s to give.
The trap on this path: answering AGENTSEC08 with a filter inventory. Answer it with a blast-radius statement instead — see lesson four.
Moving to production
AGENTOPS05 · AGENTOPS06 · AGENTPERF02 · AGENTCOST01 · AGENTCOST02
Tracing, anomaly detection and dashboards; testing and evaluation frameworks; cognitive-pipeline, model-selection and execution-path optimisation; reasoning-loop and coordination cost; model right-sizing and token consumption.
Why these five. This is the transition from it worked when I tried it to evidence you can show someone. The pairing is deliberate: AGENTOPS05 gives you per-run traces, AGENTOPS06 turns them into a pass rate, and the two cost questions are only answerable once the traces carry token counts. Do them out of order and you will be optimising cost without being able to prove you did not break quality.
The trap on this path: shipping a dashboard before a trace. Aggregate metrics you cannot drill into are decoration — you will see the pass rate fall and have no way to find out why.
Scaling to multi-agent
AGENTREL04 · AGENTPERF05 · AGENTSEC06 · AGENTCOST05
Arbiter patterns, capability taxonomies and fallbacks; orchestration and collaboration performance; inter-agent communication, trust boundaries and coordination security; cost attribution across multi-agent workflows.
Why these four. They are the four bills a second agent hands you: something has to arbitrate, coordination is serialised so latency goes up, one agent’s output is another’s instructions so the channel needs authenticating, and a single cost number for the whole system stops being actionable.
The trap on this path: running it as a design review for a system you have already decided to build. Read AGENTPERF05 first — multi-agent systems are usually slower than the single agent they replace, and the honest version of this review sometimes ends in do not add the second agent.
Hardening an existing deployment
AGENTSEC04 · AGENTSEC07 · AGENTREL06
Guardrails and human-in-the-loop for critical decisions; protecting human oversight and detecting rogue agents; integrating with legacy systems using fallbacks and idempotency.
Why these three. This is the path for a system already in production, where you cannot re-architect and the question is what to reinforce. Its distinguishing feature is that all three are about controls that decay: guardrails that were written for last quarter’s tool list, an approval queue that has grown until nobody reads it, and integrations that were fine at ten calls an hour.
The trap on this path: measuring whether the gate exists rather than whether it works. Pull the numbers — median review time and rejection rate. A gate approving 99.8% of actions in three seconds is not a control, whatever the architecture diagram says.
Run the review as a session, not a form. The value is not in the answers, it is in the argument that produces them — which means the review has to happen with people in a room (or a call) rather than as a spreadsheet circulated for completion. A form gets filled in optimistically by one person; a session surfaces the fact that two people believed different things about the same system.
Who is in the room. You need whoever can answer is that actually true in production, which is rarely the person who designed it. Four roles cover most of the 41 questions, and a review missing one of them will produce confident wrong answers in that role’s territory.
| Role | Answers which questions | What goes wrong without them |
|---|---|---|
The person who built it | AGENTOPS01–02, AGENTREL02, AGENTPERF02 — scope, prompts, tool list, the loop | Nobody knows what the agent is supposed to do, so every finding is arguable and the review drifts into design. |
The person on call for it | AGENTOPS05, AGENTOPS07, AGENTREL06–08 — traces, break-glass, degradation, legacy blast radius | The review records intended behaviour rather than actual behaviour. This is the single most common way a review produces a document that is wrong on day one. |
Someone who owns the data or the identity model | AGENTSEC01–03, AGENTSEC05, AGENTPERF07 — memory partitioning, permissions, attribution, tenancy | You get the answer the agent team believes about permissions, which is reliably broader than what is actually granted, and nobody can say what the service identity can reach. |
Someone who owns the outcome commercially | AGENTOPS06 (who signs the score), AGENTCOST05–07, AGENTSUS03 — evidence, spend, ownership | Risks get accepted by people with no authority to accept them, which means they have not been accepted at all — they have been noted. |
Evidence, not opinion. For each question, the answer is not yes — it is yes, and here is where you can see it. A trace id. A dashboard. The pull request that added the cap. The eval run and its pass rate. The runbook, and the date someone last used it. This single rule does most of the work of a good review, because show me converts an aspiration into a finding without anyone having to be confrontational about it.
Record the decision and the rejected option. The most valuable artifact a review produces is not the list of things you will fix; it is the short list of risks you are choosing to live with, in writing, with a named owner and a revisit date. We are not adding per-tenant quotas this quarter; we accept that one pathological tenant can degrade latency for everyone; revisit when we pass fifty tenants; owner: Priya. That paragraph is worth more than thirty green ticks, because in six months it is the only thing that will tell you whether the situation you are in was foreseen or was a surprise.
Which reading path fits your situation?
Interactive decision tree — outcomes:
- Building your first agent — AGENTOPS01, AGENTREL02, AGENTSEC03, AGENTSEC08
Scope, atomic tasks with minimum permissions, agent identity, input and output validation. Do these before anything else: they are the four that stop being cheap once the system is built around the wrong answer. Answer AGENTSEC08 as a blast-radius question, not a filter inventory.
- Moving to production — AGENTOPS05, AGENTOPS06, AGENTPERF02, AGENTCOST01, AGENTCOST02
Traces first, then a pass rate on a graded set, then latency and cost. In that order, because you cannot demonstrate that a cost optimisation left quality intact without the eval set, and you cannot debug the eval failures without the traces. Two teams reach this path from opposite directions — one that cannot prove quality, one that cannot explain the bill — and both need the same telemetry.
- Scaling to multi-agent — AGENTREL04, AGENTPERF05, AGENTSEC06, AGENTCOST05
Arbitration, coordination performance, inter-agent trust boundaries, per-agent cost attribution. Read AGENTPERF05 first and be willing to let it change your mind: coordination is serialised, so the multi-agent version is usually slower and always more expensive than the single agent it replaces. "Do not add the second agent" is a legitimate outcome of this review.
- Hardening an existing deployment — AGENTSEC04, AGENTSEC07, AGENTREL06
Guardrails, human oversight that cannot be talked or worn out of its job, and integrations that survive a non-deterministic caller. All three are about controls that decay, so bring numbers rather than diagrams: median review time, rejection rate, retry counts, duplicate side effects per thousand runs.
- Independent agents — start with AGENTOPS03-BP04, then a path per agent
None of the four published paths covers a growing fleet of unrelated agents, which is a real gap in the roadmap. Start with portfolio governance (AGENTOPS03-BP04, High) and decommissioning (AGENTSUS03-BP04): can you list every agent, its owner, and what data each can reach? Then run the appropriate path per agent — separately, because a fleet-wide answer averages away the one agent that matters.
Tool: Well-Architected Lens Review — Run a lens review end to end: pick a reading path for a described system, work the questions, judge the evidence you are offered, and see which risks your review would have missed.
Interactive flashcard deck.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.