Running an AI Item Through the Architecture Review Board
What actually changed in our ARB when AI arrived: the addendum, the new quorum seat, the 30-minute agenda, and the anti-patterns we had to unlearn.
When generative AI arrived, the tempting move was a new committee — an 'AI review board' with its own charter, its own cadence, its own backlog. We didn’t do that, and we would make the same call again. Our architecture review board already had the two properties a new body would take years to earn: teams were used to going through it, and it had real authority to say no. So instead of a new board, we changed three things about the existing one.
New intake questions. Two lines added to the standard submission form: does this system use AI or machine learning to produce or influence any decision or output? and does it call any model — hosted, third-party, or in-house — at runtime? A yes to either triggers the AI addendum below. The second question exists because teams genuinely did not think of 'we call an API that happens to be a model' as using AI; the first exists because a vendor product with an embedded model is still an AI system when the regulator asks.
A new quorum member. AI items cannot be heard without someone from governance/privacy in the room — a voting seat, not an observer chair. Before this rule, AI items were reviewed for scalability and cost while nobody asked what data was flowing into prompts. The quorum requirement is blunt, and blunt is what survives calendar pressure.
New artifacts demanded up front. For an AI item, the pre-read now includes a draft register entry, a data-flow diagram that shows prompts and retrieval paths as data flows, evaluation results against the intended use, and the completed addendum. No pre-read, no slot — the agenda below only works because the reading happened before the meeting.
| Addendum field | What a good answer looks like | What sends it back |
|---|---|---|
Intended purpose | One paragraph, plain language, bounded: what the system does, for whom, and what it explicitly does not do. | Purpose statements that are really ambitions — “assist employees with productivity” describes nothing and therefore permits everything. |
Legal classification | The proposed classification (EU AI Act tier, internal risk rating) with the reasoning, referencing the intake questionnaire. | A bare “minimal risk” with no reasoning. The board is not there to do the team’s classification homework live. |
Data flows — including prompts | A diagram where prompts, retrieved documents, and model outputs are first-class data flows: what enters the prompt, from where, what the model provider sees, what is logged and for how long. | Diagrams where the model is a magic box with one arrow in and one arrow out. Prompt content is a data transfer; treating it otherwise is how personal data leaks into third-party logs. |
Guardrail plan | Named controls with configuration: which Bedrock guardrail version or Content Safety severity thresholds, applied to inputs or outputs or both, and who owns threshold changes. | “We will use the platform defaults.” Defaults are a general-purpose posture, not a decision — the board wants to see that someone chose. |
Oversight design | Who reviews what, how often, with what authority to override — and how the humans stay capable of disagreeing with the system. | Oversight described as a property (“a human is in the loop”) instead of a job someone staffed and scheduled. |
Rollback plan | How the system degrades gracefully: kill switch, fallback to the pre-AI process, who may pull it, and how fast. | “We would roll back the deployment.” Rolling back software is easy; the question is what the business process does the hour after the model is gone. |
Not everything deserves thirty minutes. The board’s scarcest resource is attention, so we triage hard. Items built entirely from pre-approved parts — the enterprise LLM gateway, an already-approved model, the standard guardrail baseline, internal users only, no personal data in prompts — go on a consent agenda: approved in bulk unless a member pulls one for discussion. Items with high-risk signals skip the ARB’s authority upward: a proposed high-risk classification, personal or special-category data flowing into prompts, customer-facing autonomy, or a model or vendor we have never assessed. Everything else gets the standard slot.
Triage: consent agenda, standard slot, or escalation
- AI item submitted
- Addendum + pre-read complete?
- Returned to team
Not a rejection — a deferral. Reviewing an incomplete item burns a slot and teaches teams that documents are optional.
- Built entirely from approved patterns?
Approved gateway, approved model, baseline guardrails, internal-only, no personal data in prompts. All five or it is not a pattern match.
- Consent agenda — approved in bulk
Any member may pull an item into full review. Roughly half our AI items now land here, which is the payoff of investing in approved patterns.
- High-risk signals present?
Proposed high-risk classification, personal data in prompts, customer-facing autonomy, unassessed model or vendor.
- Escalated: risk committee + full review
The ARB still reviews the architecture, but classification and risk acceptance decisions move to the body with authority to accept them.
- Standard 30-minute slot
0–5: Purpose & classification
The chair restates the intended purpose in their own words and the team confirms or corrects — thirty seconds that regularly surfaces a mismatch between what the document says and what the team is actually building. Then the governance member states whether they agree with the proposed classification. Agreement: move on. Disagreement: the item leaves the agenda, because everything after this point depends on the answer.
5–15: Data flows & guardrails
Ten minutes on the diagram, prompts included. The questions that earn their place: what is the worst document that could end up in the retrieval corpus? what does the model provider log, and under what retention? which guardrail version is pinned, and what happens to a blocked request? On multi-cloud estates this is where platform specifics live — a Bedrock guardrail ARN, Azure Content Safety thresholds per category, and for GCP platforms the equivalent safety-filter configuration (check current docs — that stack has been renamed and re-scoped repeatedly).
15–25: Oversight & rollback
The chair asks the team to walk through one concrete failure: the model produces a confidently wrong answer to a real user on a Tuesday afternoon — narrate what happens next, minute by minute, with names. Teams that have designed oversight answer fluently; teams that wrote 'human in the loop' as a property improvise, visibly. Then rollback: who pulls the switch, what the users see, what the business process falls back to.
25–30: Decision & conditions
Approve, approve with conditions, or return. Conditions get an owner and a date and land in the tracker — a condition without a due date is a wish. The register entry is updated with the decision reference before the meeting ends, so the ARB approval and the inventory never diverge.
That agenda took a year of failure to arrive at. The failures cluster into patterns we now watch for explicitly — in ourselves as much as in the teams presenting.
The rubber stamp
Every item passes, usually with the same boilerplate conditions nobody follows up. The tell is the approval rate: if it is 100%, the board is theatre and everyone presenting knows it. Our fix was structural, not motivational — the governance seat has a real veto, conditions have owners and dates, and the board reports its returns-and-escalations count upward. A board that never says no cannot protect the teams it says yes to.
The bottleneck
The opposite failure: every AI item, however small, queues for weeks behind a monthly meeting. Teams respond rationally — they stop declaring, and the shadow inventory grows in exact proportion to the queue. The consent agenda is the antidote: invest in approved patterns so the routine items flow through in bulk, and spend the meeting only on items where judgment adds something. A review board’s throughput is a governance control in itself.
The demo that substitutes for documents
The team asks to 'just show it working' instead of submitting the addendum — and the demo is always impressive, because demos are rehearsed on curated inputs. A demo shows the happy path; the addendum forces the unhappy ones: the data flow nobody diagrammed, the rollback nobody designed. Our rule is mechanical: no addendum, no slot, and the demo happens after the documents are read, if at all. If the paperwork and the demo disagree, believe the paperwork was written to pass and the demo was built to sell.
The approval that never expires
An AI approval treated like a building permit — granted once, valid forever — while the model, the prompts, and the purpose all quietly move. Every AI approval we issue now carries a review date synced to the register’s attestation cycle, and material changes (new model, new data class, widened purpose) void it early. The system that got approved is not the system that is running eighteen months later, unless something forces the two back together.