Choosing and Evolving
Lesson 4 of 4 in Reference Architectures.
The default is not a tie-breaker — it is a strategy: start managed, evolve on evidence. The managed-API app has the lowest cost of being wrong: no capacity to size, no engine to operate, and the platform absorbs model upgrades. You graduate to a heavier architecture only when something measurable demands it — a cost model whose sustained numbers cross the self-hosting break-even, or a control requirement that a Managed endpoint cannot satisfy (weights you must own, a serving feature the platform does not expose, isolation beyond a Private endpoint).
“Evidence” has a specific shape. For cost it is the formula from Cost Modeling fed with your measured traffic — tokens per request, requests per second, peak-to-trough ratio — not a comparison of list prices at imagined volume. For control it is a requirement someone will sign, not a preference. Walk the tree with a real workload in mind.
Which architecture — and what would change the answer
Interactive decision tree — outcomes:
- Self-hosted cluster, day one
Legitimate, and expensive to do well: you take on capacity, serving engines, and safety layers simultaneously. Budget for the platform team, not just the GPUs — and keep the managed-API skeleton around it intact.
- Managed-API app + managed RAG piece
The fastest path to grounded answers: the platform runs ingest and index, you own prompts, evals, and the refresh policy. Revisit if retrieval quality plateaus below your eval bar — that is the trigger for taking over the ingest lane.
- Managed-API app + your own RAG pipeline
You keep the levers that decide retrieval quality and multi-tenant safety; the model stays managed. The corpus, chunking logic, and embeddings pipeline you build here are portable to every other architecture on this page.
- Evolve toward the self-hosted cluster
Migrate deliberately: stand the cluster up behind the same gateway contract, re-run the full eval suite against it, shift traffic gradually, and keep the managed path as fallback until the numbers hold at production load.
- Stay on the managed-API app
The right answer for most systems, most of the time. Instrument token spend and utilization now — the evidence that would change this answer only exists if you are already measuring it.
Because evolution is expected, design for it from day one by knowing what ports and what does not. The dividing line is ownership of the artifact: things that are yours — text, test cases, documents, code — move between architectures and clouds; things that are the platform’s — configuration objects, commitments, wiring — are rebuilt, not exported. The eval suite is the special one: it is both portable and the instrument that makes every other migration safe.
| Asset | Portable? | Why — the mechanism |
|---|---|---|
Prompts + templates | Mostly | Plain text in your repo — but behavior is model- and stack-specific, so every move re-runs the Eval harness before cutover. |
Eval suites + golden sets | Yes | Defined over inputs and expected outputs, provider-agnostic by construction. This is the migration safety net — the asset that proves the new stack matches the old. |
RAG corpus + chunking logic | Yes | Documents and pipeline code are yours. Changing the Embedding model forces a re-embed of the corpus — a compute cost, not a loss. |
Guardrail configurations | No | Each platform’s guardrail is its own config object with different categories, thresholds, and APIs — Bedrock Guardrails policies, Foundry content-filter configurations, Vertex safety-filter settings. The policy intent ports; the implementation is rebuilt and re-red-teamed. |
Throughput commitments + quota | No | Provisioned throughput purchases and Quota grants are term contracts with one platform, scoped to its regions and models. They wind down; they do not transfer. |
Private networking | No | PrivateLink, Private Link, and Private Service Connect are per-cloud wiring. The requirement (“no public internet path”) ports; every endpoint is re-plumbed. |
In production
These three architectures cover most LLM systems in production. Write which one you are building on page one of the design doc — reviewers can then check nodes against a known shape instead of reverse-engineering intent, and the review gets ten times easier. Per cloud, naming the architecture also names the service conversation:
AWS
Saying “managed-API app” on AWS scopes the review to Bedrock: which models from the catalog, which service tier or Provisioned Throughput commitment, which Guardrails policy version, PrivateLink or not. Saying “RAG stack” adds one question — managed or customer-managed Knowledge Base — and “self-hosted cluster” moves the conversation to EKS, EC2 GPU instance families, and who owns the serving engine. Three names, three checklists.
Azure
On Azure the architecture name maps to a deployment conversation: the managed-API app is a Foundry model deployment — which type (Global, Data Zone, or regional; standard or provisioned), which content-filter configuration is attached, Private Link or not. The RAG stack centers on Azure AI Search as the index and the grounding mechanism’s migration status; the self-hosted cluster becomes AKS plus GPU VM series and your own quota story per region.
Google Cloud
On Google Cloud the names sort the Vertex AI surface: managed-API app means a Model Garden model behind a Vertex endpoint — regional, multi-region, or global, with safety-filter thresholds and Private Service Connect settled up front. RAG stack means deciding whether RAG Engine’s corpus or your own pipeline owns ingest; self-hosted cluster means GKE, GPU machine types, and self-deployed containers in your own VPC — which Model Garden explicitly supports as a documented path.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.