Continuous Batching and Serving Engines

Why a GPU serving one request at a time is a tragedy, how continuous batching fixes it, and a map of the engines that do this for a living.

Content last verified 2026-09.

Lessons

1. Why Batching Is Everything
2. Continuous Batching
3. The Engine Landscape
4. Operating an Engine

Sources

• Kwon et al. (2023) — Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM)
• Yu et al. (2022) — Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022)
• Pope et al. (2022) — Efficiently Scaling Transformer Inference