The Latency Vocabulary

Lesson 1 of 3 in In Production: TTFT, TPOT, and the Metrics That Matter.

A generating Large language model (LLM) makes users wait twice, in two different ways. First comes a silence: the request sits in a queue, then Prefill pushes the whole prompt through every layer before a single output Token can exist. Then comes a drip: Decode produces one token per step, each step re-reading the Weights and the growing KV cache. Every latency metric in this field is a name for one of those two waits.

Time to first token (TTFT) is the silence: queue time plus prefill time (plus network, if you measure from the client — and you should). Time per output token (TPOT), also called inter-token latency, is the drip: the average gap between consecutive streamed tokens, set by decode step time. Streaming does not shorten either one — it just lets the user watch the drip instead of staring at the silence.

Put them together and the whole request is one line of arithmetic:

End-to-end latency ≈ TTFT + TPOT × output tokens.

That formula is the skeleton of every serving decision in this domain. It says a request has two independent budgets — a fixed entry fee and a per-token toll — and that they respond to different levers. Nothing you learn about engines, hardware, or cloud SKUs will matter until you can say which term your users are suffering in.

Key terms: TTFT, TPOT, Throughput, Goodput, Prefill, Decode

Latency is what one user feels. Throughput is what the system produces: output tokens per second across all concurrent requests — the number that decides how many GPUs you need and therefore what each million tokens costs you. The two pull against each other, and the next lesson is entirely about that tug-of-war.

Throughput has a failure mode worth naming now: it counts every token, including tokens nobody should be proud of. A system at saturation can post its best-ever tokens-per-second while half its requests blow their latency targets or time out and get retried. Goodput is the correction: only the output that arrives within its service-level objective counts. Tokens that show up too late — or belong to requests the client already abandoned — are work the GPU did and the product never received. When a dashboard shows throughput rising while users grow angrier, goodput is the number that would have told the truth.

Bar chart decomposing the end-to-end latency of one LLM request into three parts: queue wait 100 milliseconds, prefill 400 milliseconds, and decode 12,000 milliseconds for 400 output tokens at 30 milliseconds per token. The decode bar dwarfs the other two, showing that generation length dominates total latency for long outputs.

End-to-end latency decomposed by the formula E2E ≈ TTFT + TPOT × output tokens, for one assumed request: 100 ms queue + 400 ms prefill (TTFT = 500 ms), then 400 output tokens at TPOT = 30 ms/token → 12,000 ms of decode. Total: 12,500 ms, of which 96% is decode. The input values are generic assumptions chosen for round arithmetic; the bars are computed exactly from them. (calculated — source: Latency structure of transformer inference — Pope et al. (2022), arXiv:2211.05102)

Read the figure as a diagnosis chart. For this 400-token answer, decode is 96% of the wait — so halving TTFT, however satisfying, saves a quarter of a second, while anything that trims decode (shorter outputs, faster steps) moves the whole experience. Flip the workload and the verdict flips with it: a classifier that reads a long document and emits ten tokens lives or dies on TTFT, and no decode optimization will save it.

That is the discipline the formula buys you: profile before you optimize, and profile per term. Prompt-heavy, output-light workloads (retrieval answers, extraction, routing) are TTFT businesses — attack prefill with Prefix caching, shorter prompts, and chunked scheduling. Output-heavy workloads (drafting, code generation, reasoning models thinking out loud) are TPOT businesses — attack decode with Quantization, Speculative decoding, and hardware with more memory bandwidth. The levers barely overlap, which is why “make the model faster” is not an actionable ticket.

The end-to-end formula, worked through

Write it precisely. If a request queues for t_q, prefills in t_p, and then emits N output tokens at an average gap of TPOT seconds, the client sees:

  • TTFT = t_q + t_p (+ network overhead in client-side measurements)
  • E2E = TTFT + TPOT × (N − 1) — the first token is delivered at TTFT, so only the remaining N − 1 gaps add time; for large N everyone drops the fencepost and writes TTFT + TPOT × N.

Three workloads through the same numbers (TTFT = 500 ms, TPOT = 30 ms):

  • N = 20 (a routing decision): E2E ≈ 0.5 + 0.6 = 1.1 s — TTFT is 45% of the wait; prefill work dominates.
  • N = 400 (a chat answer): E2E ≈ 0.5 + 12.0 = 12.5 s — the figure above; decode is 96%.
  • N = 2,000 (a report): E2E ≈ 0.5 + 60.0 = 60.5 s — TTFT is a rounding error; only TPOT and output length matter.

Two honesty clauses. First, TPOT is an average over a moving target: each decode step re-reads a cache that grows with position, and under Continuous batching the batch around your request churns as neighbors join and finish — so real token gaps wobble, and serious SLOs quote TPOT as a percentile, not a mean. Second, the terms are coupled through the scheduler: the same GPU seconds serve someone’s prefill or someone’s decode, so an engine that favors incoming prompts (better TTFT) steals steps from in-flight generations (worse TPOT). That coupling is the next lesson’s subject.

In production

None of these four numbers appears on a price sheet, yet all four are measurable today from your own client — and the managed platforms increasingly speak this vocabulary back to you.

AWS

Streaming invocation on Amazon Bedrock makes the vocabulary observable from your side of the wire: time to the first streamed chunk is your TTFT, the spacing of later chunks is your TPOT — measured where it matters, with queueing and network included. Instrument both in application telemetry, split by model and by prompt class, because a managed platform’s own latency metrics describe its decode, not your user’s wait. Token quotas bite here too: a throttled-and-retried request books its retry delay into TTFT, so quota headroom is a latency feature, not just a billing one.

Azure

Azure OpenAI in Azure AI Foundry meters quota in tokens-per-minute, which ties the vocabulary to capacity planning: prompt-heavy traffic spends quota on prefill-shaped work, generation-heavy traffic on decode-shaped work, and the same requests-per-minute can be a wildly different token load. When quota saturates, the first symptom is usually rising TTFT — throttle-and-retry loops and queue growth — long before per-token decode slows. Alert on client-observed TTFT and TPOT percentiles, not on request counts.

Google Cloud

Vertex AI’s count-tokens preflight lets you decompose a slow request before blaming the model: measure the prompt in tokens, and you know whether the wait is prefill-shaped (long input) or decode-shaped (long output). Its batch/offline modes are the goodput idea productized — work with no TTFT constraint at all is scheduled purely for throughput, the cheapest tokens a GPU can make. How streamed tokens flow onward through sessions and agent frameworks is plumbing our sister AI Agent Academy covers; here the meter stops at the stream.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.