What It Costs You — Measure, Don’t Assume
Lesson 3 of 4 in Quantization.
Quantization is not free; the question is what it costs you. The uncomfortable answer: quality degradation is task-dependent and model-dependent. The same 4-bit recipe can leave casual Q&A essentially untouched while measurably hurting multi-step math, code generation, or instruction-following on edge cases — and a bit-width that one model family tolerates gracefully can hurt another, or the same family at a different size. There is no universal “4-bit costs X% quality” number, and anyone quoting one is describing their model on their Benchmark, not yours.
That leaves exactly one defensible procedure: run your own evals on the quantized artifact before shipping it — the eval suite tied to your product’s actual tasks, prompts, and languages. Building that suite is the Evaluation domain’s subject; here, the rule is just: the quantized artifact is a new model, and no new model ships without passing the same gate the old one did.
Interactive sorting exercise: Sort each claim about a quantized model: must you measure it on your own evals, or does arithmetic alone make it safe to assume? (Notice how empty the second bucket stays.)
What actually breaks: outliers and where the error hides
Why do weights quantize so much more gracefully than activations? Weight distributions are tame. Within a layer they cluster in a roughly bell-shaped bunch around zero, so a per-group scale factor (one shared multiplier for each small block of weights) spends the format’s few representable values exactly where the mass is. This is why weight-only quantization became the routine move for LLM serving.
Activations are the troublemakers. In large transformers, a handful of Hidden state channels systematically carry magnitudes far larger than the rest — the outlier channels documented across the quantization literature. A scale factor that must stretch to cover an outlier crushes the resolution left for every normal value sharing its scale; clip the outlier instead and you delete signal the network genuinely uses. That squeeze is the core reason aggressive low-bit activation quantization stays hard while 4-bit weights are commonplace.
The two methods you met in lesson two are both answers to where the resulting error should hide. AWQ’s calibration finds the weight channels that sit opposite large activations — precisely the ones whose rounding error gets amplified — and rescales them so their error shrinks, at the cost of slightly coarser representation elsewhere. GPTQ instead measures, weight by weight, how much rounding error would propagate to the layer output, and steers the error toward directions the output barely notices. Same budget of error, spent where it is cheapest — which is also why their failures are subtle and task-shaped rather than loud and obvious: the error was deliberately hidden where average behavior would not show it.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.