Continuous Batching

Lesson 2 of 4 in Continuous Batching and Serving Engines.

So batch big. The naive way — static batching — collects, say, four requests, runs them together, and returns four answers. The flaw is hiding in what generation is: each sequence stops when it emits its stop token, and output lengths vary wildly and unpredictably. One request wants a yes/no; its neighbor wants an essay. Under static batching the whole batch runs until its longest member finishes. A sequence that stops early leaves behind a dead slot — the weight stream keeps paying for it, and it produces nothing. Worse, requests arriving mid-batch must wait for the entire batch to drain before the next one forms. Measured over a realistic mix of lengths, a large share of slot-steps do no work, and the queue grows while the GPU idles through ghosts.

The fix is to stop scheduling batches and start scheduling steps. Continuous batching (also called iteration-level scheduling) makes admission and retirement decisions at every decode step: the moment a sequence finishes, its slot is freed and a waiting request takes it on the very next iteration. The batch becomes a rolling population rather than a convoy — always as full as memory and the queue allow.

Heatmap with eight rows and twelve columns. Rows one to four show static batching slots: slot one is busy for all twelve steps, slot two is busy for three steps then idle for nine, slot three busy for six steps then idle for six, slot four busy for eight steps then idle for four. Rows five to eight show continuous batching slots: each slot goes idle for at most a single step when one sequence finishes and a new request is admitted, so nearly every cell is busy. The contrast shows static batching wasting many slot-steps while continuous batching keeps the grid almost fully occupied.

Slot occupancy over twelve decode steps, static versus continuous, on a toy four-slot GPU. Static batching (top four rows) runs a batch until its longest member finishes — early finishers strand idle slots, and the next batch cannot start until step 13. Continuous batching (bottom four rows) refills each slot on the step after its sequence retires. Toy schedule with invented lengths; real engines juggle far more slots. (illustrative — source: Kwon et al. (2023) — PagedAttention / vLLM (continuous batching as deployed context))

The idea comes from Orca (Yu et al., OSDI 2022), which introduced iteration-level scheduling for transformer serving; the vLLM paper (Kwon et al., 2023) then paired it with paged KV memory, and the combination became the default architecture of modern engines. That pairing is not incidental — continuous batching needs dynamic memory. If each sequence’s KV cache had to be one contiguous block sized for the worst case, admitting a request would mean reserving the maximum context up front, and the pool would fill with reservations instead of data. Paged attention allocates cache block by block as sequences actually grow, so the scheduler can admit whenever current usage leaves room — and reclaim blocks the instant a sequence retires.

There is one more scheduling problem hiding here: a newly admitted request needs its Prefill — a burst of parallel compute over the whole prompt — while everyone else is mid-Decode. Engines handle the mix with scheduling policy: some run prefills as dedicated steps between decode iterations, some split long prefills into chunks and interleave them so ongoing streams never stall for long. The policy choice is one of the knobs behind your TTFT/TPOT trade-off, and Prefix caching softens the problem at its source by skipping prefill work for shared prompt prefixes.

Key terms: Continuous batching, Serving engine, PagedAttention, KV cache, Throughput, Goodput

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.