What the Cache Costs

Lesson 2 of 4 in The KV Cache.

One formula sizes the whole bill: KV cache bytes = 2 × layers × kv_heads × head_dim × bytes per value × tokens × batch Read it factor by factor. The 2 is one key plus one value per position. Layers each keep their own cache — a token’s keys at layer 30 are different vectors from its keys at layer 3. kv_heads × Head dimension is the width of the key (and value) vector per layer; note it counts KV heads, not query heads — a distinction Grouped-query attention (GQA) models exploit, as we will see below. Bytes per value is precision: 2 bytes at FP16 or BF16 (bfloat16). Tokens is everything in context, prompt and generation alike. And batch: every concurrent request pays its own copy. Two things about this formula should reorganize your intuitions. First, nothing in it measures how hard the text is — cache memory is bought by token count alone, so a long transcript of pleasantries costs exactly what a long legal brief costs. Second, tokens and batch multiply: long contexts and high concurrency compound into the same scarce pool of GPU memory that already holds the model’s Weights.

Bar chart of KV cache size in GiB for one request at four context lengths under a generic 32-layer model shape: 4,096 tokens is 0.5 GiB, 8,192 tokens is 1 GiB, 32,768 tokens is 4 GiB, and 131,072 tokens is 16 GiB. Growth is linear in tokens, so each 4-times jump in context multiplies the bar by 4.

Per-request KV cache versus context length for a generic teaching shape: 32 layers, 8 KV heads, head_dim 128, FP16, batch 1. For that shape the arithmetic is exact — 2 × 32 × 8 × 128 × 2 bytes = 128 KiB per token — but the shape is a stand-in, not any specific model; real architectures differ. Bars show gibibytes (GiB). (illustrative — source: Memory framing for transformer inference — Pope et al. (2022), arXiv:2211.05102)

Tool: KV Cache & VRAM Calculator — Put your own numbers in: pick a model shape, context length, precision, and batch size in the KV Cache Calculator and watch which factor dominates your memory bill.

Every factor in the formula is a lever someone has pulled. Two matter most in practice. The architectural answer: fewer KV heads. MQA and GQA share one set of keys and values across groups of query heads, cutting kv_heads — and therefore the cache — by the sharing factor, at a modest quality cost the model is trained to absorb. This decision is baked in before you ever download the weights: when you pick a model, you pick its cache footprint per token. The mechanism lives in Inside the Transformer. The serving-side answer: fewer bits per value. KV cache Quantization stores the cached keys and values at 8 bits or fewer instead of 16 — halving (or better) the bytes-per-value factor at serving time, no retraining required. The quality effect is model- and task-dependent and usually smaller than quantizing weights, but it is not free: measure on your own traffic before trusting it. A third lever — Sliding-window attention — caps how many positions are cached at all, trading away long-range attention for a bounded cache.

The formula, worked end to end

Take the generic shape from the figure — 32 layers, 32 query heads, 8 KV heads, head_dim 128, FP16. The shape is a teaching stand-in; the arithmetic below is exact for it. Per token: 2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes = 128 KiB. Per request at 8k context: 8,192 tokens × 128 KiB = 1 GiB. At 32k context: 4 GiB. Linear, exactly as the formula says. Per batch: 32 concurrent 8k-context requests × 1 GiB = 32 GiB of cache — before a single weight is counted. Now flip the levers. If this model were plain multi-head attention — 32 KV heads instead of 8 — every number above multiplies by 4: 512 KiB per token, 4 GiB per 8k request. GQA at a 4:1 grouping is a 4× cache cut; that is why nearly every recent serving-oriented architecture uses it. Quantize the cache from FP16 to 8-bit and the GQA numbers halve again: 64 KiB per token, 0.5 GiB per 8k request. Why this caps concurrency. A 7-billion-Parameter model at FP16 needs about 14 GB for weights alone (7 × 10⁹ × 2 bytes). On an 80 GB accelerator — the class of card in the verified instance table behind /tools/gpu-sizing — that leaves roughly 66 GB. At 1 GiB per 8k request, that is headroom for on the order of sixty concurrent requests in principle — less in practice, once activations, buffers, and the fragmentation problem of the next lesson take their cut. Cache memory, not compute, is usually what caps how many requests a serving GPU can hold at once.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.