The KV Cache

The memory that makes generation affordable: what gets cached, what it costs, paged attention, and prefix reuse.

Content last verified 2026-09.

Lessons

1. Why Recompute Would Kill You
2. What the Cache Costs
3. Paged Attention
4. Prefix Caching and Reuse

Sources

• Kwon et al. (2023) — Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM)
• Pope et al. (2022) — Efficiently Scaling Transformer Inference
• Vaswani et al. (2017) — Attention Is All You Need