Operating an Engine
Lesson 4 of 4 in Continuous Batching and Serving Engines.
Running an engine is not kernel engineering — the engine did that for you. What an operator owns is a small set of dials, and every one of them is a position on the same trade-off surface: weights memory versus KV pool versus per-request latency versus tokens per second. Exact flag names vary by engine; the dials do not.
Maximum concurrent sequences caps the rolling batch. The KV memory budget — usually a fraction of GPU memory the engine may claim — sets the cache pool that concurrency draws on. Weight precision decides how much of the GPU the model itself occupies: serving quantized weights (via methods like GPTQ or AWQ) frees memory that becomes cache, which becomes batch, which becomes throughput. Tensor parallel degree shards the model across GPUs — mandatory when weights outgrow one device, elective when you want more aggregate memory bandwidth and cache per replica. The speculative flag attaches a Draft model for Speculative decoding — a latency lever for low-batch regimes, largely irrelevant when the batch is already keeping the GPU busy. And maximum context length bounds the worst-case cache appetite of any single request, protecting the pool from being monopolized by one long conversation.
| Dial | Turning it up buys | And costs |
|---|---|---|
Max concurrent sequences | Throughput — more sequences amortize each weight read | Per-request TPOT creeps up; past the KV pool’s capacity, requests queue or preemption kicks in |
KV memory budget | A bigger cache pool → higher concurrency ceiling and more Prefix caching hits | Less head-room for activations and spikes; set too high, the engine risks out-of-memory failures under load |
Weight quantization | Smaller weights → more GPU left for cache → bigger batches; often also faster weight streaming | A quality delta you must measure on your own evals — never assume it is free |
Tensor parallel degree | Fits bigger models; aggregates memory bandwidth and cache across GPUs | Inter-GPU communication every layer; more expensive replicas — scaling out replicas is often the better spend for small models |
Speculative decoding | Lower per-request latency when the batch is small and bandwidth sits idle | Draft-model memory and scheduling complexity; the win shrinks as the batch — and acceptance-rate variance — grows |
Max context length | Longer documents and conversations per request | Worst-case KV appetite per sequence grows, dragging the concurrency ceiling down with it |
A capacity-planning pass
- Model + traffic profile
Inputs: parameter count, target precision, typical and worst-case context lengths, expected request rate, and latency targets (TTFT, TPOT).
- Weights memory at chosen precision
Parameters × bytes per parameter. The GPU Sizing tool computes this against the verified instance table for AWS, Azure, and GCP.
- VRAM − weights − overhead = KV pool
What the engine can hand to the cache after weights, activations, and runtime overhead take their cut.
- KV bytes per token × context = per-request cache
Cache per token depends on layers, KV heads, head size, and cache precision — the KV Cache Calculator does this per architecture.
- KV pool ÷ per-request cache = concurrency ceiling
The maximum simultaneous sequences at your typical context — the hard cap on the batch, and therefore on throughput per replica.
- Meets throughput and latency targets?
Estimate tokens/s at that concurrency, then check the queueing math against arrival rate. If the ceiling is low, the batch is small and lesson one’s economics turn against you.
- Turn a dial
Quantize weights, raise tensor parallelism or pick a bigger-memory instance, cap context length, or add replicas — then recompute the pass.
- Load-test, then ship
The paper pass gets you to the right neighborhood; only a load test with production-shaped traffic (ragged lengths, real arrival bursts) sets the final dial positions.
One warning before you optimize: the number that matters is not raw Throughput but Goodput — tokens per second that meet your latency targets. A configuration that maximizes tokens per second by running enormous batches may push every request’s TPOT past what your product tolerates; those tokens are produced but not useful. Set the latency targets first, then push the dials right until the targets start to crack. That frontier — and how to read it with TTFT and TPOT — is the subject of the capstone module.
In production
Managed model endpoints run engines like these under the hood. You rarely see the scheduler, but every dial in this lesson resurfaces as something you can buy, quota, or configure — the mechanism is identical, only the interface changes.
AWS
On Amazon Bedrock the batching machinery is invisible: you buy outcomes shaped by it — on-demand quotas metered in tokens and requests per minute, and provisioned throughput as reserved model capacity, which is someone’s batch schedule sold as units. Self-hosting on SageMaker AI or EC2 puts the actual dials back in your hands: its LLM-serving containers ship engine options and expose max-batch and parallelism settings, and your instance choice (single-GPU g-family versus multi-GPU p-family) is the tensor-parallel decision wearing an instance name.
Azure
Azure AI Foundry shows the split cleanly: serverless API deployments hide the engine and meter tokens, while provisioned deployments sell throughput units — capacity that exists because a scheduler underneath keeps its batches full. Managed compute deployments (your model, your GPU VM class) put engine flags directly in reach, and the VM-size choice against ND- and NC-series hardware is your KV-budget and parallelism decision in Azure vocabulary.
Google Cloud
Vertex AI’s prediction endpoints for self-deployed models let you pick the serving container — engines like vLLM or TensorRT-LLM among them — plus the accelerator type and count per replica: max concurrency, KV budget, and parallel degree become deployment-spec fields. Its managed model APIs hide all of that behind per-token metering and provisioned-throughput purchase, and autoscaling replica counts is the cloud’s answer to a concurrency ceiling you cannot raise on one GPU.
Tool: GPU Sizing Calculator — Run the capacity pass on real hardware specs: the GPU Sizing tool computes weights memory at each precision against verified AWS, Azure, and GCP instances, and the KV Cache Calculator fills in the per-request cache cost.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.