GQA, MoE, and Context on Real Hardware
Lesson 3 of 3 in Architecture in Production: Design Choices Set Your Bill.
Now put the whole domain on one machine. An accelerator offers a fixed pool of memory, a memory bus of fixed bandwidth, and a fixed budget of FLOPs. The Weights claim memory first — a dense model’s parameters at serving precision, or an MoE’s total parameters, idle experts included. Whatever remains is the KV budget, and it sets the number that actually governs serving economics: tokens in flight, the sum of every live session’s context. Batch size × average context must fit inside the KV budget — that single inequality is the capacity plan.
Batching is where the economics tilt. Every decode step must read the weights — but that read can be shared across all sequences in the batch, so per-token weight-traffic falls as batches deepen. The KV cache enjoys no such discount: each sequence owns its own cache, and every step reads all of them. Deep batches amortize the weights and expose the cache as the true ceiling — which is why GQA’s smaller cache converts directly into more concurrent sessions, and why MoE’s appetite for large batches (many tokens keep many experts busy) collides with the memory its total parameters already claimed.
That arithmetic is also why long-context surcharges exist, and the reason is mechanical, not commercial. A long-context request costs its host three times over. At admission, its prefill is a compute spike growing superlinearly with prompt length. For its lifetime, its cache occupies memory that would otherwise hold other sessions — an opportunity cost measured in evicted batch slots. On every decode step, its own tokens arrive slower, because each step re-reads a cache the long prompt made large — and in shared batches, stragglers complicate scheduling for everyone. Past the point where these effects dominate, a provider that did not price long context differently would be subsidizing it. Context extension stretches what the position scheme tolerates; it does not repeal the memory and bandwidth bill.
Each pressure has its own toolbox, taught in its own domain: paged and quantized caches, continuous batching, chunked prefill, and speculative decoding belong to Inference & Serving; accelerator selection, provisioned throughput, and the buy-versus-host decision belong to LLMs on the Cloud. The decision tree below routes you to the right shelf.
Which architecture pressure are you feeling?
Interactive decision tree — outcomes:
- The quadratic tax on prefill
TTFT is prefill time: the whole prompt must pass through every layer before one token can stream, with attention work growing superlinearly on long prompts. Levers: shorter or cached prefixes (prompt/context caching), chunked prefill so long prompts don’t stall other requests, compute-rich hardware for the prefill phase. Serving-side techniques live in the Inference & Serving domain; managed-API mitigations (prompt caching, batch modes) in LLMs on the Cloud.
- Weight-footprint pressure
Your memory is spent before the first user arrives: total parameters × serving precision exceeds the pool — MoE totals being the classic surprise, since memory bills on total parameters while compute bills on active ones. Levers: weight quantization, tensor or expert sharding across devices, or a smaller (or denser) model. Sizing and instance-topology guidance lives in LLMs on the Cloud.
- KV-cache pressure
Memory that grows with sessions and context is the cache’s signature. Levers: fewer KV heads (GQA/MQA — an architecture choice made at training time, so often a model-selection criterion for you), cache quantization, paged cache management, context limits and eviction policy. Serving-side cache management is taught in Inference & Serving.
- The good kind of busy — tune the scheduler
High steady concurrency is what accelerators are built for: continuous batching amortizes weight reads across sequences, and MoE models get the token variety their experts need. Your lever is scheduler and batching quality, not architecture. Throughput tuning lives in Inference & Serving.
- Utilization pressure — maybe don’t host at all
Spiky, low-concurrency traffic leaves self-hosted accelerators idle but paid for, and batches never get deep enough to amortize anything. A per-token managed endpoint moves utilization risk to a provider who pools it across customers. The buy-versus-host arithmetic is worked through in LLMs on the Cloud.
In production
Three pressures — weight footprint, KV occupancy, and batch economics — are the whole story of LLM serving cost, on every cloud, under every pricing scheme.
AWS
Self-hosting, the tokens-in-flight inequality is the capacity plan: weights (total parameters, at precision) come off the top of instance memory, and what remains, divided by the per-token cache cost, is the ceiling on concurrent context. Long-lived sessions are memory tenants, so idle timeouts and context limits are capacity tools, not just product choices. On managed endpoints, quotas denominated in tokens per minute are the provider rationing the same three pressures, pooled across customers.
Azure
Reserved-capacity deployments make the mechanism unusually visible: what a unit of provisioned throughput rations is, underneath, a slice of memory for tokens in flight plus the bandwidth to re-read them every step. Context length is the hidden multiplier — doubling average context roughly halves the sessions a reservation can hold at the same latency, whether or not any pricing table says so explicitly.
Google Cloud
Long-context tiers on managed endpoints track real per-step cost: a bigger cache read on every decode step and a longer memory tenancy per request. If your workload is prefill-heavy and latency-tolerant, batch or offline modes recover utilization by packing prompts; if it is chat-heavy, architectures with small caches (fewer KV heads) and accelerators with high memory bandwidth are what the tokens-per-second-per-dollar arithmetic rewards.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.