Platform team, product teams: what to centralize

Lesson 2 of 5 in AgentOps Org Patterns: Maturity, Team Shapes, and Gates That Scale.

Once more than two teams ship agents, the org chart becomes an engineering decision. The shape that works is the one that worked for cloud infrastructure and CI: a central agent platform serving product teams. The platform owns the runtime, the guardrail catalog, the eval infrastructure, and the tool registry. Product teams own the agent’s job, its judgment, and its consequences.

Both extremes fail in a recognisable way. With no platform, five teams build five harnesses, five injection filters, and five trace formats — and during an incident nobody can answer “which of our agents can send email?” With too much platform, every prompt change queues behind a platform sprint, so product teams quietly build their own runtime and you get shadow agents the kill switch does not reach.

Replit, July 2025 — why the fix was a platform fix

SaaStr founder Jason Lemkin publicly documented that Replit’s AI coding agent deleted his production database despite an explicit code freeze it violated repeatedly; Replit’s own output called it “a catastrophic error of judgement,” it wrongly told him rollback was impossible (the rollback in fact worked), and it had fabricated data including a 4,000-record database of fictional people. Lemkin said he had told it not to act “eleven times in ALL CAPS.”

The instructive part is the response. CEO Amjad Masad called the incident “Unacceptable and should never be possible” and announced automatic development/production database separation to prevent it “categorically,” plus staging environments and forcing documentation search. Eleven capitalised instructions are a prompt; an environment boundary the agent cannot reach across is a mechanism — and mechanisms of that kind are built once, by a platform, for every agent. That division is the whole argument of this lesson.

Sources: The Register, 2025-07-21 and 2025-07-22.

What to centralize, what to federate — and the symptom when the line moves
Concern	Platform owns	Product team owns	Symptom when the line is wrong
Runtime / harness	The loop, checkpointing, timeouts, retries, idempotency keys, trace emission, tenant isolation.	Nothing — consume it. Contribute upstream when it lacks something.	Federated: the same timeout bug gets fixed four times, and one team never hears about it.
Guardrail catalog	The implementations — injection classifier, PII redaction, egress allowlist, output-schema validation — plus the org-wide floor nobody may waive.	Which optional guardrails to enable, and thresholds above the floor for their risk profile.	Federated: each team writes its own regex; a bypass found in one codebase is never fixed in the other four.
Evals: infra vs content	The runner, trace store, judge harness, CI wiring, and the signed score record.	The golden set, the rubrics, and the pass threshold for their task.	Centralized content: the platform writes generic cases that pass while real customers fail.
Tool / MCP registry	The registry, the allowlist gate, version pinning, an owner and review date per entry, egress scope per tool.	Requesting entries, justifying the scope they need, and the business logic of tools they author.	Federated: servers installed per project from a package registry; during an incident there is no inventory to query.
Prompts and context	Templates, context-assembly libraries, prompt-versioning conventions, context budget tooling.	The actual words — role, tool guidance, escalation rules, definition of done.	Centralized: prompt edits queue behind a platform sprint, so teams fork the runtime in secret.
Model access and cost	Provider contracts, the gateway, rate limits, per-team quotas, cost attribution by tag.	Model choice within the approved set, and their own cost/quality trade-offs.	No gateway: a surprise invoice with no per-agent attribution. Over-central: nobody can try a cheaper model without a ticket.
Observability	Trace schema, retention, PII handling, the alerting plumbing, cross-agent dashboards.	Which alerts matter for their agent’s task, the thresholds, and the runbook behind each one.	Centralized alerts: pages fire on CPU and queue depth and miss the agent confidently doing the wrong thing.
Kill switch	The mechanism and its scopes — one agent, one tool, one tenant, everything — plus an audit trail of who flipped what.	The decision to flip it for their agent, and the drill that proves they can.	Either way wrong: if flipping needs a platform engineer or a deploy, it is not used during the incident that needs it.
On-call	Runtime pages: harness down, queue stalled, provider outage, cost guardrail tripped.	Behaviour pages: task success rate, wrong actions, cost per task, escalation quality.	One shared pager: platform on-call cannot judge whether an output was wrong, and the product team never learns its own failure modes.

One more design rule, borrowed from platform engineering: build a paved road, not a fence. The platform path must be the fastest way to ship an agent, not the compliant-but-slow way. If a product team can reach production sooner by writing its own loop, some of them will, and those are precisely the agents missing from your inventory when you need to disable a tool everywhere.

That makes platform adoption a measurable thing rather than a mandate: what fraction of production agents run on the platform runtime, and what fraction of tool calls go through the registry? A platform at 60% adoption has a 40% blind spot, and blind spots are where incidents live.

Anti-pattern: the Center of Excellence that reviews everything

A weekly board reviews every agent change. Within two months the queue is three weeks deep, and teams learn to describe changes as “minor prompt tweak” to skip it — so the board reviews only the changes that did not matter.

Fix: replace calendar review with trigger-based review (next lesson), and make the paved road pass the review by construction — an agent using only registry-approved tools and platform guardrails should need no board at all.

Anti-pattern: the platform with no customers

Nine months of platform building in isolation, launched to product teams who already shipped on their own harnesses. Now there are two runtimes and the platform team is asking for a migration budget.

Fix: one real customer before the second feature. Ship the thin slice — trace emission, a tool registry, a kill switch — and let the first team’s incidents tell you what to build next.

Anti-pattern: N teams, N harnesses

No platform at all. Five eval harnesses, five injection filters, five trace formats, no org-wide inventory. The tell arrives during the first incident: nobody can answer which of our agents have write access to the CRM? in under a day.

Fix: you do not need a full runtime to fix this. Start with the two cheapest cross-cutting artifacts — an agent inventory (owner, tools, autonomy tier, data reached) and a shared kill-switch mechanism — then grow the runtime under the teams that want it.

Anti-pattern: security as the only gate

The security review is the sole quality bar, so it inherits questions it was never designed to answer. It correctly flags a prompt-injection exposure and has no opinion about a 30% task failure rate — and because it is the only gate, it becomes the bottleneck everyone routes around.

Fix: layered gates with different owners. The eval gate answers “does it work?” and is owned by the product team plus the platform mechanism. Security review answers “what can it reach, and who can make it do something?” and fires on triggers, not on every release.

Anti-pattern: the platform owns the prompts

Well-intentioned centralisation: prompts are risky, so the platform team reviews and stores them. Now the people who know the refund policy cannot change the refund agent’s escalation rule without a ticket, and the platform team is guessing at domain judgment it does not have.

Fix: platform owns prompt versioning, templating, and diffing; the product team owns the words. Centralize the mechanism, federate the judgment.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.