The Throughput–Latency Frontier
Lesson 2 of 3 in In Production: TTFT, TPOT, and the Metrics That Matter.
The batching module left you with a fact and a temptation. The fact: Decode is memory-bandwidth-bound, so a GPU serving one request at a time wastes almost all of its arithmetic — Continuous batching fills the idle compute by decoding many sequences against each shared read of the Weights. The temptation: if batching is nearly free throughput, why not batch harder, forever?
Because each admitted sequence taxes every other one. A bigger batch means more KV cache contending for memory and bandwidth, more work per scheduler iteration, and — past the point where compute or bandwidth saturates — longer decode steps for everyone. Throughput climbs toward a hardware ceiling while per-token latency climbs without one. Plot every achievable operating point and you get a curve: the throughput–latency frontier. You cannot be above it; you choose where to sit on it. Interactive products sit left (small batches, snappy tokens, expensive GPUs); offline pipelines sit right (fat batches, slow individual streams, cheap tokens). “Fast and cheap” is not a point on the curve.
Line chart with two curves showing inter-token latency in milliseconds (log scale) versus system throughput in tokens per second. As batch size grows from 1 to 128, throughput rises from about 90 to about 1,400 tokens per second and then flattens, while median latency rises from 11 to 90 milliseconds and p99 latency rises from 14 to 400 milliseconds, diverging sharply from the median near the throughput ceiling.
Two production rules fall straight out of this picture.
Percentiles, not means. The p99 curve lifts away from the median long before the ceiling: at high load, one request in a hundred is living a completely different — and much worse — life than the typical one. A mean would average the two stories into a number that describes neither, and it mixes prefill-shaped and decode-shaped delays into one meaningless figure. Real SLOs are written per metric, per percentile: p50 TTFT below X, p99 TTFT below Y, p99 TPOT below Z. The p99 matters more than its one-percent name suggests — heavy users make many requests, so nearly all of them meet your tail eventually, and timeouts, retries, and abandoned sessions all live out there.
Saturation is a cliff, not a slope. The frontier shows steady-state batches; add arrival traffic and queueing theory takes over. While offered load is comfortably below capacity, queues stay short and TTFT barely notices. As load approaches capacity, queue length — and with it TTFT — grows explosively: the same system that absorbed 10% more traffic at half load can see waits multiply near full load. A system running “hot” at 95% utilization is not efficient; it is one traffic blip away from a TTFT incident. Capacity planning means choosing an operating point left of the knee, with headroom bought on purpose.
| Symptom | What it means on the frontier | The lever that fits |
|---|---|---|
p99 TTFT spikes at peak hours; p50 barely moves | Queueing at saturation — offered load is crossing the knee, and the unlucky tail absorbs the wait | Add capacity or shed load before the knee: scale out, admission control, route overflow elsewhere |
TPOT degrades for everyone as concurrency rises | Operating point slid right — batches grew until decode steps lengthened for all sequences | Cap concurrent sequences or batch size; accept lower throughput per GPU as the price of the SLO |
Throughput plateaus while latency keeps climbing | The hardware ceiling — compute or memory bandwidth is saturated; extra admissions only add queueing | More or better hardware, or shrink the work itself: Quantization, shorter outputs, Speculative decoding |
Long prompts stall other users’ token streams | Prefill bursts monopolizing scheduler iterations — the TTFT/TPOT coupling made visible | Chunked prefill and scheduler policy — engine knobs from the batching module |
In production
Every serving option you can buy is a position on someone’s frontier. The real question behind each cloud mechanism is: whose frontier, and who picks the operating point?
AWS
On shared serverless endpoints such as on-demand Bedrock, the provider runs the frontier and chooses the operating point for a multi-tenant fleet; your latency percentiles include everyone else’s traffic, and your protections are quota headroom, retry budgets, and timeouts you set deliberately. Self-hosting on EC2 GPU instances hands you the knobs directly — engine limits on concurrent sequences and batched tokens choose your point on the curve. Load-test with your own traffic shape to find your knee, then autoscale on token throughput and queue depth, not request count: requests are not the unit the GPU feels.
Azure
Provisioned throughput is the frontier rented whole: a reserved slice of model-serving capacity that only your traffic fills, which is precisely why its latency is steadier than shared serverless — no neighbors, no mystery operating point. The mechanism to plan around is spillover: when your reserved capacity saturates, overflow can route to shared token-billed capacity. That is admission control as a product decision — you are choosing which requests keep the reserved-capacity SLO and which take their chances on the shared frontier.
Google Cloud
The same split shows up as Vertex AI’s pay-as-you-go endpoints versus provisioned throughput, and — for self-hosters — GKE clusters running an open serving engine. Kubernetes adds the operational half of the lesson: autoscaling signals should be frontier-aware (token throughput, queue depth, batch occupancy) because CPU-style signals miss GPU saturation entirely, and scale-out must trigger left of the knee — by the time p99 TTFT is screaming, the queue explosion has already happened and new replicas arrive late to the incident.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.