What You Inherit
Lesson 2 of 3 in Managed APIs vs Self-Hosting.
Cloud engineers already know the shared-responsibility model for infrastructure. Model serving has its own version, and each stop on the spectrum redraws the line. LLMs in Production gave you the first look at this trade; here is the full ledger.
The mistake to avoid is binary thinking — managed means they handle it, self-hosted means we handle it. Every stop leaves a set of responsibilities on your side of the line, and some of them (evaluation, guardrail configuration, quota headroom) never leave your side no matter how managed the platform is. The interesting question at each stop is not whether you own things, but which things just became yours.
| Stop | You own | Provider owns |
|---|---|---|
Serverless token API | Prompts and context; evaluation and Regression testing across model versions; guardrail / Content filter configuration; Quota headroom and 429 handling; version pinning ahead of deprecations; cost monitoring | Hardware, serving engine, and scaling; model hosting and patching; multi-tenant isolation; the deprecation calendar (their schedule, not yours) |
Provisioned throughput | Everything above, plus capacity planning: how many units, which term, and the Utilization of the commitment you now pay for around the clock | Still the entire serving stack — the same managed API, now with throughput the provider guarantees |
Managed compute endpoint | Everything above, plus the model artifact itself: which weights, which container image (where supported), instance sizing, scaling policy, and paying for idle | Endpoint provisioning and operation; host management; hardware lifecycle and physical capacity |
Self-managed on Kubernetes | Everything above, plus the serving stack: Serving engine upgrades, GPU drivers and device plugins, CVE patching, autoscaling behavior, observability, and on-call | The VMs or managed Kubernetes control plane; physical security; the accelerator supply chain |
Two rows in that ledger move the most, and they deserve names.
The model lifecycle line. On serverless, the provider updates and retires model versions on their calendar. That is mostly a gift — security patches and quality improvements arrive without a deploy — but it means a model you depend on can be deprecated out from under you. Your defense is on your side of the line: pin versions where the platform allows it, and keep an Eval harness ready so a forced migration is a regression-test run, not a scramble. Deploy your own weights at stops 3–4 and the lifecycle inverts: nobody will ever retire your model, and nobody will ever patch it either.
The quota line. At every managed stop, your real ceiling is not the hardware — it is a Quota. Bedrock controls inference with token-usage quotas viewable in the Service Quotas console, with a documented increase-request process. Azure assigns tokens-per-minute per subscription, per region, per model, per deployment type, and rejects over-limit requests with HTTP 429. Vertex AI enforces per-project, per-region, per-base-model request quotas, with tuned models sharing the base model’s quota. Managed capacity is real capacity — but it is allocated, and the allocation is a production dependency you must monitor like any other.
Quota mechanics worth knowing before an incident
A few documented behaviors that surprise teams at 2 a.m.:
Separate meters can hide behind one model. Bedrock’s two inference endpoints, bedrock-runtime and bedrock-mantle, each have separate per-model quota allocations — traffic to each is tracked independently even for the same model. The docs also note default quotas may be adjusted per account based on regional factors, payment history, or approved requests, so two accounts can see different ceilings for identical workloads.
Bursting exists, but only opportunistically. Azure’s dynamic quota (documented as a preview) lets standard deployments exceed their configured TPM when spare capacity exists. It can save a spike — and it can vanish exactly when the region is busiest, which is when you needed it. Plan launches on guaranteed capacity, not on bursts.
Batch draws from a different pool. Azure batch deployments consume a separate enqueued-token quota so bulk jobs do not disrupt online traffic; Vertex AI documents batch inference for Gemini as having no predefined quota at all, drawing on a dynamically allocated shared pool. Moving offline work to batch tiers is therefore also a quota-management move, not just a price move.
Committed capacity is the documented escalation path. When on-demand quotas cap you, all three clouds point the same direction: Bedrock Provisioned Throughput in Model Units, Azure provisioned deployments in PTUs, and Vertex AI Provisioned Throughput covering baseline traffic with spillover falling back to pay-as-you-go.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.