What the Specs Mean

Lesson 2 of 4 in GPUs and Accelerators.

A GPU spec sheet is a wall of numbers, but LLM serving only asks four questions of it — and each question is answered by a different line.

Memory capacity bounds what fits. The model’s weights at serving precision, plus a KV cache entry for every token of every concurrent sequence, plus runtime overhead, must fit in accelerator memory: roughly weights_bytes + kv_bytes + overhead ≤ capacity. Quantization shrinks the weights term; longer context windows and bigger batches grow the KV term. The KV Cache Calculator puts your numbers into that inequality.

Memory bandwidth bounds decode speed. Generating each token means streaming the active weights (and the sequence’s KV cache) from memory through the compute units — so at low batch, tokens per second is a bandwidth number, not a FLOPs number. This is the Prefill/Decode split from the inference domain wearing hardware clothes.

Compute bounds prefill. The prompt is processed in one parallel pass with plenty of arithmetic per byte read, so TTFT on long prompts is where FLOPS earns its keep.

Interconnect bounds multi-GPU serving. When one model spans several GPUs, Tensor parallelism synchronizes the GPUs at every layer of every step. NVLink-class fabrics (the menu marks them: “SXM” variants, Azure ND-series “NVLink” notes) are built for that chatter; PCIe-attached GPUs (Azure NC-series “PCIe” variants) pay far more for it. The menu encodes this in names — now you can read it.

From serving requirement to the spec that governs it

  1. What do you need from the box?

    Every serving SLO traces back to one line on the spec sheet. Work the requirement first, then shop for the number that governs it.

  2. Model + context must fit

    Weights at serving precision + KV cache for all concurrent sequences + runtime overhead. Quantization shrinks the first term; context length and batch size grow the second.

  3. Fast first token (TTFT)

    Prefill processes the whole prompt in one parallel pass — lots of arithmetic per byte read, so compute throughput is what binds on long prompts.

  4. Fast streaming (tokens/sec per user)

    Decode emits one token per sequence per step and must re-read the active weights each step. At low batch there is little math per byte — bandwidth binds.

  5. Model larger than one GPU

    Tensor parallelism splits every layer across GPUs and synchronizes at every step. The interconnect carries that traffic thousands of times per generated token.

  6. Governed by: memory capacity

    Use the provider-documented GB for the instance (lesson 1), never nominal chip figures.

  7. Governed by: compute (FLOPS)

    The one line where raw compute rules — and only until the prompt is processed.

  8. Governed by: memory bandwidth

    The spec that usually rules inference — see the deep dive below for why.

  9. Governed by: interconnect class

    NVLink-class (SXM, ND-series) vs PCIe-attached (NC-series, some single-GPU boxes) — a qualitative but decisive menu distinction.

Why bandwidth, not FLOPS, usually rules inference

Put named variables on the two ceilings a decode step faces. Let W be the bytes of active weights, K the KV-cache bytes read for the sequences in flight, B the accelerator’s memory bandwidth (bytes/s), F its usable compute (FLOP/s), and b the batch size.

Each decode step must move roughly W + K bytes through the compute units, so its duration has a memory floor of (W + K) / B. The same step performs on the order of 2 × params × b FLOPs, giving a compute floor of 2 × params × b / F. Which floor is higher decides which resource binds. At small b, the compute floor is tiny — a handful of FLOPs per weight byte moved — while the memory floor is fixed: every step re-reads the model no matter how few tokens it produces. Modern accelerators ship vastly more FLOP/s than bytes/s, so the memory floor wins by a wide margin. That is why decode-phase Model FLOPs utilization (MFU) is famously low, and why a spec-sheet FLOPS comparison between two instances can invert once you measure tokens per second.

The same algebra explains the serving toolbox. Continuous batching raises b, so each weight read is amortized across more sequences — arithmetic intensity climbs toward the compute floor, which is how Throughput-oriented deployments earn their money. Quantization shrinks W, directly lowering the memory floor per step. Speculative decoding gets several tokens verified per full-model read. Every one of these is a way of paying the bandwidth bill less often — none of them buys FLOPS, because FLOPS was never the constraint.

Tool: KV Cache & VRAM Calculator — The fit inequality with your numbers in it: layers, heads, precision, context, and batch in — weights plus KV cache out, next to real instance capacities.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.