Why Batching Is Everything
Lesson 1 of 4 in Continuous Batching and Serving Engines.
From the KV cache module you know what one Decode step is: read the current state, attend over the cached keys and values, and push one Token through every layer. Here is the part that sets up this whole module: to produce that one token, the GPU must stream essentially every weight of the model from GPU memory to its compute units. Each matrix multiply reads its full weight matrix and applies it to a single vector.
Count what that costs. Moving a Parameter from memory earns you roughly one multiply-add of useful work — a couple of FLOPs per weight fetched. But a modern accelerator can execute hundreds of FLOPs in the time it takes to fetch those bytes. At batch size 1, the arithmetic units finish their share almost instantly and then wait on memory for the rest of the step. The GPU is not computing; it is remembering. Compute utilization during single-stream decode is a rounding error, and you are paying for the whole chip.
Now add a second request. The weights are already streaming — attention aside, the same weight read can serve both sequences, one matrix-vector product each. Thirty-two requests? Still roughly one weight read per step. The memory traffic that dominates the step is nearly constant in batch size, so decode throughput scales almost linearly with batch — nearly free — until something else runs out. That something is memory: every admitted sequence brings its own KV cache, and when the cache pool is full, admission stops.
Line chart with batch size from 1 to 128 on the x-axis and relative performance on the y-axis. One line, decode throughput, rises steeply and near-linearly at small batch sizes, bends gradually at larger ones, and stops at batch 128 where KV cache memory runs out. A second flatter line, per-token latency per request, rises only slowly across the same range. The gap illustrates that batching multiplies total tokens per second at modest per-request cost until memory is exhausted.
This is the single most important economic fact about serving LLMs. Cost per token is the price of the GPU-hour divided by the tokens it produces in that hour — so a batch that multiplies Throughput by some large factor divides cost per generated token by nearly the same factor, on the same hardware, running the same model. Providers that serve millions of users are not doing anything you cannot do; they are running the batch dial far to the right. Conversely, a self-hosted deployment that serves requests one at a time pays list price for a GPU and uses a sliver of it. That is the batch = 1 tragedy: the hardware bill is fixed, and every empty slot is money burned.
The ceiling is concrete. Weights claim their share of GPU memory first; what remains is the KV cache pool; each concurrent sequence consumes cache in proportion to its context length. Long contexts mean fewer simultaneous sequences, which is why architectures that shrink the cache — Grouped-query attention (GQA), Multi-query attention (MQA), Sliding-window attention — are really batch-size features, and why the KV cache module called the cache the scarcest resource in serving.
The roofline arithmetic, symbolically
Put symbols on the intuition. For a dense model with N parameters stored at p bytes each (p = 2 for BF16 (bfloat16)), one decode step for one sequence reads about N·p bytes of weights and performs about 2N FLOPs of matrix work — one multiply and one add per parameter. Arithmetic intensity, the ratio of work to traffic, is therefore about 2N / (N·p) = 2/p ≈ 1 FLOP per byte at 16-bit precision.
Accelerators are built expecting orders of magnitude more work per byte than that; their peak-FLOPs-to-bandwidth ratios run into the hundreds. The machine is unbalanced for this workload: at intensity ~1, the time per step is set entirely by bytes moved, and the compute units idle. Batching B sequences reuses each weight fetch B times: intensity rises to roughly 2B/p, and the step stays memory-bound — with throughput growing almost linearly in B — until B is large enough to approach the hardware balance point.
Two costs do grow with B and eventually bend the curve. Attention reads each sequence’s own KV cache — that traffic is per-sequence and cannot be amortized across the batch — and activation memory grows with B as well. In practice the binding constraint arrives earlier and blunter: the KV pool simply fills. Quantizing weights shrinks N·p and hands the freed memory to the cache, which is why quantization shows up in this story as a batch enabler, not just a smaller download.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.