QLoRA: Training on a Budget

Lesson 3 of 4 in Fine-Tuning: Full, LoRA, QLoRA.

LoRA removed the optimizer-state multiple, but look at what is left on the GPU: the frozen base weights themselves, sitting in 16-bit precision, now the largest single slice of training memory. They receive no updates — they exist only to be read during the forward and backward pass. QLoRA’s question (Dettmers et al., 2023) is blunt: if we are never going to change these numbers, why store them so generously?

So QLoRA stores the frozen base in 4-bit — using NF4, a 4-bit data type the paper designed around the observation that pretrained weights are approximately normally distributed, so quantization levels should be spaced for a normal distribution rather than evenly. The trainable LoRA adapters stay in higher precision (BF16 (bfloat16)), and during training, gradients flow through the dequantized 4-bit weights into the adapters. Two supporting tricks round out the paper: double quantization (quantizing the quantization constants themselves, shaving more memory) and paged optimizers (spilling optimizer state to CPU memory during spikes instead of crashing). The headline result: fine-tuning a 65-billion-parameter model on a single 48 GB GPU while matching the task performance of full 16-bit fine-tuning — work that previously required a cluster, reduced to one card. That is the paper’s claim and its legacy: QLoRA moved fine-tuning of large open-weight models from lab budgets to single-GPU-class hardware.

The QLoRA training setup

  1. 16-bit base checkpoint

    The pretrained open-weight model as downloaded — the input to the whole procedure.

  2. Quantize to 4-bit NF4, freeze

    A one-time conversion. NF4 spaces its 16 levels for normally distributed weights; double quantization compresses the per-block scaling constants too.

  3. Forward pass: dequantize on the fly

    Weight blocks are expanded back to BF16 just long enough to do each matrix multiply — the model computes in 16-bit; only storage is 4-bit.

  4. LoRA adapters in BF16

    The bypass matrices from the previous lesson, attached in higher precision. They are the only trainable parameters in the system.

  5. Loss on the training demonstrations

    Standard SFT loss — QLoRA changes where the numbers live, not what is being learned.

  6. Backward pass through frozen 4-bit weights

    Gradients propagate through the dequantized base into the adapters. The 4-bit weights themselves receive no updates — 4-bit has nowhere near the resolution for gradient steps.

  7. Optimizer updates adapters only

    Adam state exists only for the tiny adapter matrices; paged optimizers park it in CPU memory during spikes.

Keep the two precisions straight, because they answer different questions. The 4-bit base answers “how do I fit training in memory?” — it is a storage format for numbers that never change. The BF16 adapters answer “where does learning happen?” — gradient descent needs resolution, so the moving parts stay in a format with room to move. The asymmetry is the design: aggressive compression exactly where nothing needs to change, full precision exactly where everything does.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.