The Managed-API App
Lesson 1 of 4 in Reference Architectures.
Most LLM systems in production are one of three shapes. A managed-API app calls a hosted model through a Managed endpoint and owns everything around the call. A RAG stack adds a document pipeline so answers ground in your data. A self-hosted cluster replaces the hosted model with Serving engine pods on GPUs you rent. Everything else — agents, batch pipelines, fine-tuned specialists — is a variation on one of these skeletons.
This lesson draws the first and most common shape. The model call itself is one node; the architecture is everything you build around it so that one node can be trusted with production traffic. Read the diagram top to bottom, and open each node — the details name the per-cloud pieces.
The managed-API app, end to end
- Your application
Web, mobile, or backend service. It owns timeouts, retries with backoff, and streaming to the user — and it never holds model credentials. Everything model-shaped happens behind the gateway.
- API gateway + auth
Authenticates callers, enforces per-client rate limits, and logs every request. Centralizing here means one place to meter Token spend per tenant and one choke point when a client misbehaves — instead of every app burning the shared platform Quota directly. Each cloud has managed gateway services; the pattern matters more than the product name.
- Prompt assembly
System prompt, few-shot examples, user input, and any retrieved context are composed under a Context window budget. Keep the static parts (system prompt, tool definitions) at the front of the prompt — that ordering is what makes the cache node below earn its keep.
- Guardrails: input
Platform-level screening of the assembled prompt. AWS: Amazon Bedrock Guardrails — content filters (including a Prompt Attack category), denied topics, word filters, and sensitive-information filters for Personally identifiable information (PII); applied by guardrail ID and version, or standalone via the ApplyGuardrail API. Azure: built-in content filtering powered by Azure AI Content Safety, with optional Prompt Shields for direct and indirect Prompt injection; a blocked prompt returns HTTP 400. GCP: Vertex AI safety filters — non-configurable blocks (CSAM, SPII) plus configurable harm-category thresholds.
- Prompt / prefix cache
Managed platforms can reuse computation for repeated prompt prefixes — the mechanism is Prefix caching: the KV cache state for tokens the platform has already processed is reused instead of recomputed, cutting TTFT and often the metered price of those tokens. Support and discount rates vary by platform and model — check current docs for the models you deploy.
- Managed model API
The hosted model behind a Managed endpoint, chosen from the platform’s Model catalog. AWS: Amazon Bedrock — on-demand invocation with documented service tiers, or Provisioned throughput purchased in Model Units. Azure: a Foundry model deployment — serverless standard, provisioned, or batch types, with Global and Data Zone variants routing across regions (Cross-region inference). GCP: Vertex AI — Standard/Priority/Flex pay-as-you-go or Provisioned Throughput in GSUs. On all three, traffic can stay off the public internet via a Private endpoint (AWS PrivateLink, Azure Private Link, GCP Private Service Connect).
- Guardrails: output
The completion is screened before your code touches it. AWS: Bedrock Guardrails evaluates responses too, including contextual grounding checks against source material. Azure: completions run through the same Content filter classifiers; a filtered completion is signaled with a content_filter finish_reason your code must handle. GCP: configurable content filters score responses by probability and severity per harm category, and the API reports block reasons via enums.
- Output handling
Your code, treating model output as untrusted input: validate structure against a schema, strip or sandbox anything executable, apply business rules, and fall back gracefully — a degraded answer beats a stack trace. This is where the security domain’s single-channel lesson becomes code.
- Response to user
Streamed where latency matters — the user sees tokens as they decode. The stream must still pass through output handling; buffering just enough to validate is a real design tension.
- Monitoring + evals
Traces, token counts, latency percentiles, guardrail-block rates — plus an Eval harness running scored samples of real traffic. This node closes the loop: prompt changes ship through Regression testing, and Drift in quality or cost pages a human before users do.
Notice what the diagram is really doing: every earlier domain of this curriculum is a node. The architecture is not new knowledge — it is the placement of knowledge you already have. That is also how to review one: walk the nodes and ask which team owns each, what its failure mode is, and where its config lives.
| Node | Domain that owns it | What it settles | Go deeper |
|---|---|---|---|
Prompt assembly | Adaptation | How prompts become engineering artifacts | |
Guardrail layers | Security | What guardrails can and cannot stop | |
Prompt / prefix cache | Inference | The KV-cache mechanics that make prefix reuse cheap | |
Managed model API | This domain | Consumption modes, catalogs, and throughput per cloud | |
Output handling | Security | Why output is untrusted input from the same channel | |
Monitoring + evals | Evaluation | Building evals that gate production changes |
Key terms: Managed endpoint, Provisioned throughput, Private endpoint, Cross-region inference, Quota, Utilization
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.