Fewer Bits: What Quantization Is
Lesson 1 of 4 in Quantization.
Every Parameter in a model is a number stored in some format, and the format decides two things: how many bytes the number occupies and which values it can represent exactly. A 16-bit float has 65,536 distinct bit patterns spread cleverly across a huge range. A 4-bit integer has sixteen. That is the entire mechanism of quantization: re-express the same Weights in a format with fewer bits, rounding each original value to the nearest one the small format can represent.
Fewer representable values means every weight lands slightly off its trained value — quantization is deliberate, systematic rounding error. The bet is that a network’s behavior is mostly robust to small nudges in its weights: the Large language model (LLM) you get back is the same model, almost. The word “almost” is what lesson three is about; this lesson is about why the bet is so tempting.
The temptation is arithmetic. Weight memory is simply parameter count × bytes per parameter, so every halving of bytes halves the largest cost of hosting a model — before you change anything else about it.
Bar chart showing bytes per parameter for four number formats: FP32 at 4 bytes, FP16 or BF16 at 2 bytes, 8-bit formats at 1 byte, and 4-bit formats at 0.5 bytes. Each halving of bytes halves the weight memory of the same model.
“Quantize the model” is actually three decisions, made independently. Weights are quantized offline, once, producing a new artifact — the easiest win, because weights are frozen and their distributions are known in advance. Activations — the values flowing through the network during a forward pass — are quantized on the fly if at all, and they are harder (lesson three explains the outlier problem). The KV cache is its own pool of memory that grows with every token in context; quantizing it is a separate lever with its own quality trade, covered from the memory side in The KV Cache. Serving stacks routinely quantize weights aggressively while leaving activations and cache in 16-bit — “weight-only” quantization.
| What gets quantized | What it is | When it happens | What it buys you | Difficulty |
|---|---|---|---|---|
Weights | The frozen parameters on disk and in GPU memory | Offline, once — produces a new artifact | Smaller GPU (or more room for cache and batching); fewer bytes read per Decode step | Easiest — distributions are static and known |
Activations | Values flowing through layers during the forward pass | At runtime, every request | Lower-precision matrix math end to end | Hardest — outlier channels resist low bits (lesson 3) |
KV cache | Cached keys/values, growing with every token in context | At runtime, as entries are written | More concurrent long-context requests per GPU | Middle — a quality trade that scales with Context window use |
Tool: GPU Sizing Calculator — Run the arithmetic yourself: pick a model size and a bytes-per-parameter format in the GPU sizing tool and watch which GPUs the weights fit on.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.