The Serving Loop

Lesson 3 of 3 in Prefill and Decode.

Strip away the dashboards and a serving endpoint runs the same dance for every request. The text is tokenized into IDs. A scheduler finds room for the request — on a busy Serving engine it typically joins a batch already in flight (Continuous batching, its own module). Prefill runs once, writing the KV cache and producing the first token; if the prompt shares a prefix with earlier traffic, Prefix caching can skip part of that work. Then the decode loop spins: forward pass, Logits, a token chosen under whatever Temperature and sampling settings the caller sent, append, check the stop conditions, emit. When a stop fires, the server closes the stream and reports usage — input tokens and output tokens, metered separately, exactly as the two phases incurred them.

One request through a serving endpoint

  1. Request arrives

    Prompt plus generation settings: sampling parameters, max_tokens, stop sequences, stream on or off.

  2. Tokenize

    Text becomes token IDs. Input token count — the first half of the bill — is fixed here, before the model runs.

  3. Schedule into a batch

    The engine admits the request when there is memory for its KV cache. Time spent waiting here lands in TTFT.

  4. Prefill: one parallel pass

    The whole prompt is processed at once; the KV cache is written; logits for the first output token come out the end.

  5. Sample the next token

    Logits become probabilities; one token is chosen under the request’s sampling settings. Covered in Decoding and Sampling.

  6. Stream the token out

    Each loop iteration yields exactly one token, so the server can forward it immediately — this is where streaming UX comes from.

  7. Stop condition met?

    Three tripwires: the model samples its end-of-sequence token, the max_tokens budget runs out, or a caller-supplied stop sequence matches.

  8. Decode: one forward pass

    The newest token goes in, the KV cache is read and appended, next-token logits come out. One TPOT tick per trip around this loop.

  9. Close stream, report usage

    Final counts of input and output tokens — the bill mirrors the two phases exactly.

Streaming is not a feature bolted on top — it is the decode loop made visible. The model produces exactly one token per pass, so the server can hand each one to the client the moment it exists; buffering the full response adds nothing but silence. This is why every chat UI types: TTFT decides how long the screen stays blank, TPOT sets the typing speed, and a smooth drip of tokens is users watching your decode loop in real time.

The stop conditions deserve respect, because they are the only things standing between you and an unbounded bill. An Instruction-tuned model learns to emit an end-of-sequence token when its answer is done — the graceful exit. max_tokens is the circuit breaker: when the budget runs out mid-thought, the text just stops, which is why a response cut off mid-sentence is usually a budget stop, not a model failure. Stop sequences let the caller define custom endings — invaluable for structured output, dangerous when a chosen string can occur in normal prose. (How agent frameworks consume these streams — sessions, tool-call deltas, reconnects — is the sister AI Agent Academy’s territory; here, the mechanism ends at the stream.)

In production

The prefill/decode asymmetry is platform-independent physics. Every serving stack meters and tunes the two phases differently, which is why a long-prompt-short-answer workload and a short-prompt-long-answer workload behave like different applications on the same model — everywhere you run it.

AWS

Amazon Bedrock meters input and output tokens separately and streams responses token by token, so both the bill and the API surface the two phases directly. A retrieval-heavy workload (big prompts, terse answers) spends its budget in the cheap input column but pays in TTFT; conversational and code-generation traffic lives in decode, where per-token pace rules the experience. Self-hosting on EC2 GPU instances makes the same physics your operations problem: decode throughput tracks the instance’s memory bandwidth, which is why the GPU sizing tool reasons from bandwidth and memory, not just FLOPs.

Azure

Azure OpenAI in Azure AI Foundry denominates quota in tokens per minute, and the asymmetry means identical quota supports very different request mixes: prompt-heavy traffic burns quota in prefill bursts, while completion-heavy traffic holds serving capacity through long decode loops. Latency objectives on Azure-hosted models should always be written as two numbers — time to first token and tokens per second of streaming — because a single end-to-end figure hides which phase is failing you.

Google Cloud

Vertex AI exposes a count-tokens API, which means the prefill side of a request is measurable before you send it — but the output side is a model decision until a stop condition fires, so honest cost estimation treats output length as a distribution, not a constant. Streaming endpoints surface the decode drip directly, and capacity planning has to consider workload shape (input-heavy versus output-heavy), not just total token volume.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.