The Engine Landscape
Lesson 3 of 4 in Continuous Batching and Serving Engines.
Everything in the last two lessons — per-step scheduling, paged cache memory, prefill/decode interleaving — is code somebody has to write, tune, and keep correct under concurrency. That somebody is a Serving engine: the specialized server that stands between an HTTP request and a GPU. Loading weights into a framework and calling generate() in a loop is a demo; an engine is what makes the same model a service.
Five jobs define the species. An engine schedules — continuous batching, admission control, queue priorities. It manages KV memory — paged allocation, Prefix caching, eviction when the pool tightens. It executes the model fast — fused GPU kernels for attention and matrix work, quantized-weight support, tensor-parallel execution across multiple GPUs. It implements the decoding zoo — Temperature, Top-p (nucleus) sampling, stop sequences, structured output, Speculative decoding when configured. And it speaks a protocol — an HTTP API with streaming, plus the metrics (TTFT, TPOT, queue depth) an operator needs. Miss any one of the five and the others are academic.
A vertical stack diagram of five layers. From bottom to top: GPU kernels and runtime, described as fused attention and matrix kernels; model executor, described as weights, quantized formats, tensor parallelism; KV cache manager, described as paged allocation and prefix reuse; scheduler, described as continuous batching and admission control, and visually emphasized; API frontend, described as OpenAI-compatible HTTP, streaming, and metrics.
A handful of open projects dominate the landscape, and they are best understood by what each one optimizes for — not by benchmark numbers, which shift with every release, every model, and every workload shape. Treat any published engine-versus-engine speed claim as a snapshot of one configuration on one day; if the choice matters to you, benchmark your model on your traffic.
| Engine | Notable for | Ecosystem |
|---|---|---|
vLLM | The project that popularized PagedAttention plus continuous batching as a package; broad model coverage and a long feature list (prefix caching, quantized formats, speculative decoding, multi-GPU) | Open source with a large community; the common default for self-hosting open-weights models on GPUs, and a frequent substrate under managed offerings |
TensorRT-LLM | NVIDIA’s engine: models compiled into optimized kernels for NVIDIA hardware, with aggressive kernel-level tuning and quantized-precision support | Open source but NVIDIA-centric by design; commonly paired with a separate inference server frontend in enterprise NVIDIA stacks |
SGLang | Fast structured generation and program-style multi-call workloads; known for aggressive KV reuse across requests that share prefixes (its RadixAttention idea) and for constrained/JSON output | Open source, research-born like vLLM; popular where applications chain many calls over shared context — agent and pipeline workloads |
TGI (Text Generation Inference) | Hugging Face’s production server: continuous batching, streaming, quantized loading, with first-class integration into the Hub’s model ecosystem | Open source, maintained by Hugging Face; a natural fit where teams already live in the Hugging Face toolchain |
llama.cpp | CPU-first (plus consumer-GPU) inference in plain C/C++ with the GGUF quantized file format; runs models on laptops, phones, and edge boxes with no Python stack | Open source with an enormous hobbyist and edge community; the default answer to “run it locally”, and the origin of many small-footprint deployment patterns |
What “OpenAI-compatible” implies mechanically
Nearly every engine above exposes an endpoint shaped like the OpenAI chat-completions API, and that phrase carries real mechanical commitments — it is a serving contract, not a marketing label.
The server owns text processing. The client sends a JSON list of chat messages; the engine applies the model’s chat template (roles, System prompt, special tokens) and runs the Tokenizer. Two engines serving the same weights with different templates are effectively serving different models — a classic source of quality mysteries when teams switch engines and quietly change the template.
Sampling parameters map onto the decode loop. temperature, top_p, max_tokens, stop in the request body become the Sampling configuration for that request inside the shared batch. Every request in the batch can carry different settings; the engine applies them per sequence at the Logits stage each step.
Streaming mirrors decode. With stream: true, the connection stays open and the engine emits server-sent-event chunks as decode steps complete. The first chunk’s arrival is your TTFT, and the inter-chunk gap is your TPOT — the protocol makes the serving phases directly observable from the client side.
Usage is token metering. The usage object reports prompt and completion token counts — the engine counting Prefill and Decode work per request, the same meter managed platforms bill on.
The payoff of the shared shape: swap the base URL and existing SDKs, gateways, and eval harnesses point at your own engine unchanged. The trap: compatibility covers the request surface, not behavior — template details, default stop handling, and how servers treat unsupported parameters differ across engines, so contract tests beat assumptions.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.