Formats and Deployment

Lesson 4 of 4 in Quantization.

Quantized weights arrive from three directions, with very different trust profiles. Vendor-provided: the model’s publisher ships official quantized variants alongside the full-precision Checkpoint — quantized with a known recipe, usually evaluated by the people who trained the model. Do-it-yourself: you take the FP16/BF16 checkpoint and run GPTQ, AWQ, or a GGUF conversion yourself, choosing the method, bit-width, and calibration data — maximum control, and now the eval burden is entirely yours. Community: Open weights hubs host third-party quantizations of popular models in many formats and bit-widths — enormously convenient, but you are trusting someone else’s recipe, calibration choices, and upload discipline.

Whichever route, format decides what can load it — qualitatively: GGUF artifacts belong to the llama.cpp ecosystem; GPU serving engines such as vLLM, SGLang, TGI, and TensorRT-LLM each document which quantized checkpoint styles they accept, with GPTQ- and AWQ-format artifacts widely supported across them. Check your engine’s current docs before you standardize on a format; support matrices move faster than any lesson can.

From FP16 checkpoint to deployed quantized artifact

  1. FP16/BF16 checkpoint

    The released full-precision weights — the reference your quantized artifact will be judged against.

  2. Pick method + bit-width

    GPTQ, AWQ, a GGUF scheme… and how many bits. Engine compatibility constrains the menu.

  3. Pick calibration data

    A small sample that resembles real traffic. Calibration statistics steer where rounding error lands.

  4. Quantize → new artifact

    An offline job producing a new set of weights — a different model from the checkpoint it came from.

  5. Run YOUR evals on the artifact

    The product’s own eval suite, against the exact artifact — not the FP16 original, not someone else’s benchmark.

  6. Quality within budget?

    Compare against the full-precision baseline on the metrics your product actually lives by.

  7. Pin the artifact: hash + recipe

    Record the content hash, method, bit-width, calibration set, and tool config. This tuple is the model.

  8. Deploy to a compatible engine; monitor

    Roll out like any other release — staged, monitored, revertible to the pinned previous artifact.

Close the loop on the meters. Smaller weights leave more GPU memory for the KV cache, which means more concurrent requests and bigger batches — higher Throughput from the same card (size the cache with the KV cache calculator). And because Decode is dominated by reading the weights from memory for every generated token, weight-only quantization cuts the bytes read per step — the mechanism by which it can speed up generation as well as shrink the footprint. Or spend the savings the other way: the same model on a smaller, cheaper GPU. Either way the lever is the same arithmetic from lesson one — which is exactly why it is the first lever every self-hosting team reaches for.

In production

Quantization is the biggest single lever on self-hosted serving cost — bytes per parameter decide the smallest GPU that can host a model and how much room is left for cache and batching. The same lever is a silent quality regression if artifacts are not versioned.

AWS

Self-hosting on GPU instances or SageMaker endpoints, the weight format sets which instance families can hold the model at all, and how much memory remains for KV cache and batch depth — the throughput side of the bill. Store quantized artifacts in versioned, hash-addressed storage, and promote a new quantization exactly like a new container image: through staging, behind your eval gate, with instant rollback to the previous pinned artifact.

Azure

On AKS GPU node pools or managed endpoints in Azure AI Foundry, the same arithmetic picks your node SKU family, and a quantized variant can be the difference between one-GPU and multi-GPU hosting. Register each quantized artifact as an immutable model version in your registry — method, bit-width, and calibration recorded as metadata — so a deployment references a version, never a mutable name.

Google Cloud

On GKE or Vertex AI endpoints, treat the quantized artifact as its own registered model version and canary it against the full-precision baseline on mirrored traffic before cutover — the cleanest way to catch task-specific degradation your offline evals missed. The GPU-memory arithmetic decides accelerator choice here too; the artifact registry decides whether you can prove which weights served which request.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.