Two Phases, Two Personalities
Lesson 1 of 3 in Prefill and Decode.
Send one request to an Large language model (LLM) and the hardware lives through two very different phases. First the model reads your entire prompt — system message, conversation history, retrieved documents, all of it — in one big parallel pass: every prompt Token flows through the Transformer stack at once, and the Attention keys and values computed along the way are written into the KV cache. That is Prefill. It ends the moment the model produces Logits for the first output token.
Then the personality flips. Generation is Next-token prediction in a loop: sample a token, append it, run one full forward pass to get the next distribution, repeat. That is Decode — one token per pass, and each pass needs the previous token as input. No amount of parallel hardware can produce token 200 before token 199 exists. (The dependency is so fundamental that a whole technique family — Speculative decoding — exists just to cheat it. Later module.)
The two phases stress the machine in opposite ways. Prefill does an enormous amount of arithmetic on weights it just loaded — matrix-matrix multiplies spanning the whole prompt — so it is typically compute-bound: the GPU’s arithmetic units are the ceiling. Decode does a sliver of arithmetic per pass but must re-read the model’s Weights (plus the growing cache) from memory for every single token, so it is typically memory-bandwidth-bound: the ceiling is how fast bytes move, and the arithmetic units mostly idle.
Key terms: Prefill, Decode, TTFT, TPOT, KV cache
Bar chart comparing time spent in the two phases of one illustrative request. The prefill bar, covering all 1,000 prompt tokens in one parallel pass, is 400 milliseconds. The decode bar, covering 200 output tokens generated one pass at a time, is 6,000 milliseconds — fifteen times longer, showing that generation dominates wall-clock time.
This split is not trivia — it is where the two numbers on every serving dashboard come from. Time to first token is essentially the price of prefill, plus any time the request spent queued: the user stares at an empty screen while the whole prompt is processed. Time per output token is the price of one decode pass: it sets the pace at which text streams once it starts. Double the prompt and TTFT roughly doubles while the streaming pace barely moves; ask for twice the output and TTFT is untouched while total time balloons. The two meters move independently because they measure different machines wearing the same GPU — the domain capstone, In Production: TTFT, TPOT, and the Metrics That Matter, builds your whole latency vocabulary on this split.
| Trait | Prefill | Decode |
|---|---|---|
Tokens per forward pass | The entire prompt at once | One — the newest token only |
Parallelism | Across all prompt positions — matrix-matrix math | None across time — each pass waits for the last token |
Typical bottleneck | Compute (FLOPs) | Memory bandwidth — weights re-read for every token |
KV cache role | Writes it: one entry per prompt token | Reads all of it, appends one entry per step |
Latency meter it drives | TTFT — time to first token | TPOT — time per output token |
Why decode is bandwidth-bound: the arithmetic-intensity argument
The ratio that decides which resource limits a workload is arithmetic intensity: floating-point operations performed per byte moved from memory. Modern accelerators can execute far more FLOPs per second than they can stream bytes from memory, so a workload with low FLOPs-per-byte leaves the compute units idle, waiting on loads.
A decode step is the low-intensity extreme. Producing one token costs roughly two FLOPs per Parameter (a multiply and an add) — but every parameter must be read from memory to contribute. Two operations per parameter read is far below what the hardware could sustain, so the step time is set by bytes, not math: to a first approximation, one decode pass cannot finish faster than (bytes of weights + bytes of KV cache read) ÷ memory bandwidth. That is why per-token pace barely notices prompt length but responds directly to anything that shrinks bytes moved — smaller weights via Quantization, shared KV heads via GQA, a leaner cache (size yours with the KV Cache Calculator).
Prefill escapes because it amortizes. Processing 1,000 prompt tokens in one pass reads the weights once and applies them to 1,000 positions — a thousand times the arithmetic per byte. Intensity climbs past the hardware’s balance point and the workload turns compute-bound. Serving engines pull the same trick on decode by batching many requests, so each weight read serves many tokens at once — the Throughput economics behind Continuous batching, covered in its own module. Pope et al. (2022) develop this memory-versus-compute accounting for transformer inference in detail (arXiv:2211.05102).
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.