The mandate and the three operating models
Lesson 2 of 6 in The AI Center of Excellence: Organizing for AI Adoption.
Everything an AICoE does flows from one design decision, so get it right before hiring anyone: the mandate is dual. The center exists to enable — find the valuable use cases, build the shared platform, teach the skills, ship pilots that prove value — and to govern — set the standards, tier the risks, run the gates, keep the registry honest. Microsoft’s Cloud Adoption Framework captures both halves in one sentence: an AI CoE is 'an internal team of experts who drive successful and valuable AI outcomes' that simultaneously 'prevents fragmented or ungoverned AI adoption'.
Drop either half and the center fails in a predictable direction. Govern-only becomes the gatekeeper — a review queue with no delivery muscle, which teams learn to route around (Microsoft’s agent-adoption guidance names the gatekeeper as the first of three CoE failure modes, and its Azure guidance is blunter still: 'replace the CoE gatekeeper model that blocks work with an advisory group that sets guardrails'). Enable-only becomes a skunkworks — cool demos, no standards, and the CAF-AI whitepaper’s warning made flesh: 'a common mistake is to evolve AI units that do not deliver on business value.' The working posture is accelerator with brakes: the same team that helps you build is the team that tells you what good looks like — which is precisely why its advice gets taken.
With the mandate fixed, the structural question follows: where do the people sit? Three operating models cover the field. You met the same three shapes in the last module as governance archetypes — where the authority to say no lives. Here the question is different: where does the delivery and enablement capacity live — who builds, who advises, who staffs the use cases. Microsoft’s agentic-CoE guidance offers the same menu (centralized, hybrid, federated), and both vendors agree on the trajectory more than the starting point.
| Dimension | Centralized | Hub-and-spoke | Federated |
|---|---|---|---|
Who builds AI solutions | The CoE builds nearly everything itself | The hub builds shared platform + hardest cases; trained spokes in each unit build the rest on hub standards | Business units build; the center is a thin advisory forum and standards library |
Expertise required outside the center | Almost none — that is the point | A trained governance-and-delivery lead per major unit | Deep AI skill in every participating unit |
Speed and scale | Fast for the first ten use cases, then the queue forms | Scales with the number of trained spokes | Fastest locally — if the skills are real |
Consistency risk | Lowest — one team, one standard | Managed — hub calibrates spokes through shared rubrics and QA | Highest — standards drift unit by unit without strong platform enforcement |
Typical stage | Year one: skills scarce, trust unbuilt (Microsoft: early-stage companies benefit from a centralized CoE) | Years two to three: demand outgrows the center | Maturity: governance embedded in platform, CoE shifts to advisory (Microsoft’s explicit end-state) |
The models are stages, not tribes. The common migration runs centralized → hub-and-spoke → federated-with-guardrails, and the trigger points are observable: Microsoft’s guidance tells CoEs to watch for approval delays, knowledge bottlenecks, and rising friction between product teams and the center — the signs that centralization has gone from concentrating scarce skill to rationing abundant demand. The healthy response at each inflection is to push capability outward and keep standards central: train spokes, embed governance into the platform so it enforces itself, and let the center retreat to the work only a center can do. Migrating too early is just as real a failure — a federation declared before the spokes exist is decentralization wearing a nicer name, and you built the shadow-AI spiral an org chart.
One more boundary matters: the AICoE is a neighbor, not a replacement. Microsoft’s agentic guidance is explicit that a CoE 'doesn’t replace your existing governance' — it adds AI-specific capability on top of it. Map the borders on day one, because turf ambiguity kills more CoEs than incompetence does.
AICoE vs the AI governance board
The board (or committee — its design fills the operating model module) decides: contested cases, risk acceptance, policy approval. The CoE prepares and executes: it runs intake and triage, briefs the board with evidence, then carries decisions back into delivery. Clean test: the board can exist for years without building anything; the CoE builds every week. In many organizations the CoE also provides the board’s secretariat — fine, as long as the decision rights stay with the board.
AICoE vs the architecture review board (ARB)
The ARB owns technical soundness across all systems: integration patterns, security architecture, tech-stack choices. The AICoE owns the AI-specific review layer — evaluation quality, model risk, data fitness — and publishes the AI patterns the ARB then treats as approved architecture. In practice the CoE sends a member to ARB sessions that touch AI, and the ARB fast-tracks solutions built on the CoE’s pattern library.
AICoE vs the cloud CoE
The CCoE owns landing zones, cloud guardrails, and platform operations — the rails the AICoE’s workloads run on (the cloud guardrails module covers these). Microsoft’s recommendation: if a CCoE exists, integrate AI practice into it rather than standing up a rival. Where the AICoE is separate, the division of labor is payload vs platform: the CCoE keeps the infrastructure safe; the AICoE keeps what runs on it evaluated, tiered, and registered.
AICoE vs the data office
The chief data office owns data quality, lineage, catalogs, and permissible use — the fuel line. The AICoE is the data office’s biggest customer and its most demanding one: every AI use case starts with a data-fitness question. The working interface is shared machinery, not shared ownership: the data office’s stewards sit in the CoE’s intake reviews, and the CoE’s registry links every system to the datasets it consumes.
Choose your operating model
Interactive decision tree — outcomes:
- Start centralized
Scarce skills plus concentrated demand is the textbook case for a centralized CoE — Microsoft’s guidance says early-stage adopters benefit most from consolidation. Build credibility with delivered use cases, and plan the spoke-training program now: the queue that ends this model is roughly a year away.
- Hub-and-spoke
You have (or can train) capable people in the units, and enough risk exposure that standards must stay centrally owned. The hub keeps the rubric, the platform, and the hardest reviews; spokes run local intake and delivery. Invest in calibrating the spokes — drift between them is this model’s failure mode.
- Federated with guardrails
Skills are broad, the platform enforces the standards, and trained leads own delivery locally — the center can shift to the advisory end-state Microsoft describes: guardrails and forums rather than gates and queues. Keep the registry, the standards, and the calibration reviews central; everything else belongs in the units.
- Centralized — with a countdown
Broad demand, manual enforcement, high regulatory stakes: centralize now so risk decisions stay consistent, but treat it as triage, not a destination. Every month in this state, the queue grows and shadow AI compounds. Your two urgent investments: platform guardrails that automate enforcement, and spoke training that adds capacity.
- A lean center and a community of practice
With deep distributed skills, low regulatory stakes, and narrow demand, a heavyweight CoE would be bureaucracy in search of a mission. Run a minimal center — registry, standards library, office hours — and a community of practice. Revisit the moment demand broadens or a high-risk use case appears.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.