Sizing for LLMs: memory, GPU, token capacity
Lesson 2 of 3 in Model Deployment and Throughput Strategies.
Container-based FM deployment is its own skill because LLM serving breaks every intuition from classic ML deployment. A tabular model is megabytes and answers in one forward pass; an LLM is tens or hundreds of gigabytes of weights that must fit in GPU memory before the first token, answers by generating one token at a time, and holds a KV cache per in-flight request that grows with context length. So an LLM endpoint is sized along three axes at once — memory, GPU utilization, and token processing capacity (tokens per second, not requests per second) — and deployed through container-based patterns built for exactly that: on SageMaker AI, the Large Model Inference (LMI) containers, AWS's specialized deep learning containers for LLM serving.
A vertical stack diagram of GPU memory contents in an LLM serving container, bottom to top: model weights as the dominant resident block; the KV cache above it, growing with context length multiplied by concurrent requests; activations and framework overhead as the thin top layer. The lesson: weights set the floor, the KV cache sets how much concurrency fits above that floor.
Memory
Weights set the floor: a model's parameter count × precision must fit in the fleet's combined GPU memory before a single request is served. The levers: quantization (lower-precision weights — the same model in less memory, with a quality trade to validate) and tensor parallelism (shard the model across multiple GPUs and run them as one). The KV cache then claims what remains — which is the real limit on concurrency.
GPU utilization
Token-by-token generation leaves naive GPUs idle between small steps. Continuous batching fixes it: the server interleaves many requests' generation steps, admitting new requests into the batch as others finish — the single biggest throughput feature in LLM serving stacks. It is why a purpose-built LLM container outperforms a generic one on identical hardware.
Token capacity
Plan in tokens per second, not requests per second. One analytics request with a 90k-token context can cost more than a hundred chat turns. Capacity math starts from measured input/output token mix — the same unit Provisioned Throughput prices with MUs, which is no coincidence: tokens are the native unit of FM capacity everywhere.
Loading strategies
The AIP-C01 exam guide's "specialized model loading strategies" are about the gigabytes between S3 and GPU memory: store artifacts uncompressed (SageMaker AI supports deploying uncompressed models, skipping a long untar on every scale-up), stream weights to the device, and remember that LLM endpoints auto scale slowly — a new replica must pull and load the full model before it serves, so scale-up is minutes, not seconds. Provision headroom accordingly.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.