The Cloud Serving Menu
Lesson 3 of 3 in In Production: TTFT, TPOT, and the Metrics That Matter.
Back in Foundations you met the hosting fork as a first look: managed API, managed platform, or your own GPUs. You now own the vocabulary to reread that menu the way a serving engineer does. Every option on every cloud is the same machinery — Prefill, Decode, KV cache, Continuous batching — wrapped in a different answer to one question: who owns the frontier?
Serverless token APIs put the provider’s frontier behind a per-token price. You never see the batch scheduler; you feel it — as quotas (requests-per-minute and tokens-per-minute), as burst limits, as latency percentiles that include every other tenant’s traffic. Provisioned or dedicated throughput rents you a private slice of the frontier in capacity units: each unit is, mechanically, a reserved allocation of accelerator memory, bandwidth, and compute, rated for some token throughput on a specific model. Self-hosting a Serving engine on GPU instances hands you the whole curve: engine choice, batching knobs, hardware, and every consequence of getting them wrong.
| Shape | Billing unit | Who owns the frontier | The limits you actually hit | Wins when |
|---|---|---|---|---|
Serverless token API | Tokens (input and output metered separately; cached input often discounted) | The provider — operating point, engine, hardware all invisible to you | Quotas: requests/min and tokens/min; burst throttling (retry-after); shared-fleet latency variance | Spiky or low traffic, prototypes, anything that must scale to zero; you pay nothing for idle |
Provisioned / dedicated throughput | Capacity units per unit time — billed while idle, sized in advance | Shared: the provider runs the engine, but only your traffic fills the slice, so your shape sets the operating point | The capacity you bought: saturate your units and you are the noisy neighbor; overflow needs a spillover plan | Steady, high volume with strict tail-latency SLOs; launch events; predictable unit economics |
Self-hosted engine on GPU instances | Instances per hour (plus the team that runs them) — tokens are free only in the sense that nobody meters them for you | You — engine, batch limits, Quantization, hardware, autoscaling: the full frontier and its knee | VRAM (weights + KV cache), memory bandwidth, and your own capacity planning; every quota is one you must invent | Custom or fine-tuned Open weights models, hard data-control requirements, or measured economics that beat per-token prices at your volume |
Two mechanisms on this menu deserve a closer reading, because both are this domain’s physics wearing a product name.
Capacity units are the two meters, bundled. A provisioned-throughput unit reserves bytes (accelerator memory for Weights and cache) and FLOPs-plus-bandwidth (prefill and decode work) — which is why providers size and rate these units in tokens per unit time under an assumed prompt/output shape, and why two workloads with identical request counts can consume wildly different numbers of units. Sizing a reservation is a frontier exercise: measure your peak token throughput, input and output separately, then buy units so the peak sits left of the slice’s knee. Sizing from request counts is the classic failure.
Serverless quotas are someone else’s admission control. Tokens-per-minute limits, burst ceilings, and retry-after responses are the provider protecting their frontier’s knee from you — the same levers you would deploy self-hosting, run in reverse. Treat quota headroom as part of your latency budget, and build backoff-and-retry as a first-class citizen, because at peak, a throttled request is a TTFT incident by another name.
Comparing the shapes for a real workload is a numbers exercise, and two of this site’s tools do the arithmetic: the Cloud Model Matrix maps which models are consumable in which shape on which cloud, and the Token Cost Estimator turns your measured traffic — tokens in, tokens out, peak concurrency — into comparable costs across shapes. For the self-hosted branch, the GPU Sizing tool and KV Cache Calculator estimate whether your model and traffic even fit the instances you are eyeing.
From workload to serving shape
Interactive decision tree — outcomes:
- Serverless token API
Pay per token on the provider’s shared frontier. Scale-to-zero economics and zero serving operations; in exchange you inherit quotas, burst limits, and latency percentiles shaped by tenants you cannot see. Engineer for it honestly: retry with backoff, measure client-side TTFT/TPOT percentiles, keep quota headroom as latency budget — and use batch/offline lanes for work with no TTFT constraint. Revisit when traffic becomes steady and high; this is a door, not a wall.
- Provisioned / dedicated throughput
Rent a private slice of the frontier in capacity units — reserved accelerator memory, bandwidth, and compute rated for token throughput on your model. Your traffic alone sets the operating point, so tail latency stabilizes and unit economics become predictable. The honest trades: you pay while idle, sizing is your responsibility (from peak token throughput and prompt/output shape, never request counts), and saturating your own slice needs a spillover plan.
- Self-hosted engine on GPU instances
You own the whole curve: engine, batching limits, quantization choices, hardware, autoscaling, and the knee nobody else will find for you. Maximum control over weights, data path, and cost structure — and the full operational bill: load testing under your real traffic shape, capacity planning, incident response. Choose this because constraints or measured economics demand it, and budget the team, not just the GPUs.
In production
All three clouds sell all three shapes. The names differ; the mechanism underneath — who owns the frontier, and what unit meters it — is the same everywhere.
AWS
On-demand Amazon Bedrock is the serverless shape: token-metered, account-level quotas, shared capacity, with batch modes for throughput-shaped work. Provisioned throughput on Bedrock sells the capacity-unit mechanism directly — units of reserved model-serving capacity, each rated for token throughput, billed while idle, sized by you. The self-hosted shape runs on EC2 GPU instances (or SageMaker endpoints as a middle rung), typically with an open serving engine such as vLLM or TensorRT-LLM; the instance catalog in this site’s GPU tools is the menu, and the KV-cache arithmetic from earlier modules is the sizing math.
Azure
Azure AI Foundry makes the menu explicit per deployment: standard (serverless, token-billed, tokens-per-minute quota) versus provisioned-throughput deployments, where a provisioned unit is the capacity-unit mechanism with a nameplate — reserved model-processing capacity you size against measured token throughput and prompt/output shape. Spillover between the two is the admission-control pattern from the frontier lesson, productized. The self-hosted shape lives on GPU virtual machines or AKS, where every frontier knob — and every consequence — is yours.
Google Cloud
Vertex AI runs the same three rungs: pay-as-you-go endpoints with token quotas at one end, provisioned throughput reserving generative capacity for latency-sensitive traffic in the middle, and Model Garden weights deployed to your own GKE GPU infrastructure at the far end. GCP illustrates the recap cleanly because the same open-weights model can often be consumed in all three shapes — same weights, same physics, three owners of the frontier — which makes the choice legible as operations and economics, never as model capability.
And that closes the domain. Inference & Serving began with one request splitting into Prefill and Decode; everything since — Sampling knobs, the KV cache and PagedAttention, Context window budgets, Quantization, Speculative decoding, Continuous batching — has been a way to move one of the numbers this capstone named: TTFT, TPOT, Throughput, Goodput, and the bill. When a serving decision looks murky, come back to the two questions this module drilled: which term of the end-to-end formula is hurting? and who owns the frontier?
You can now run a model well. The next domain, Adapting LLMs, is about making it yours — prompting, retrieval, fine-tuning, and distillation, and the engineering judgment of choosing between them. The serving meters do not switch off there: every adaptation choice you will meet, from stuffing context to training a smaller specialist, is also a move on the latency and cost dials you just learned to read.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.