LoRA: Low-Rank Adaptation
Lesson 2 of 4 in Fine-Tuning: Full, LoRA, QLoRA.
LoRA — low-rank adaptation, from Hu et al. (2021) — starts from one empirical bet: the change a fine-tune needs to make to a weight matrix is much simpler than the matrix itself. So don’t update the matrix. Freeze it, and learn the change as a separate, deliberately skinny pair of matrices bolted on beside it: a down-projection A into a tiny intermediate dimension, and an up-projection B back out. Their product BA is the update the layer would have learned — expressed with a fraction of the parameters, because the intermediate dimension, the rank r, is small.
That intermediate dimension is the method’s capacity dial. Higher r means a more expressive update and more trainable parameters; lower r means cheaper and more constrained. The LoRA paper’s striking finding was how far down the dial can go: very low ranks matched full fine-tuning on many tasks — against GPT-3 scale models it reported cutting trainable parameters by roughly four orders of magnitude, with GPU memory during training cut to roughly a third.
The LoRA bypass around a frozen weight matrix
- Input activations x
The activations flowing into an adapted layer — in the paper, attention projections such as the query and value matrices.
- Frozen pretrained W₀
The original weight matrix. It receives no gradients and no optimizer state — everything the base model knows stays untouched.
- A: project down to rank r
Trainable. Maps the input into a tiny r-dimensional space. Initialized with small random values.
- B: project back up
Trainable. Maps back to the layer’s output dimension. Initialized to zero, so at step 0 the bypass adds nothing and the model behaves exactly like the pretrained base.
- Add the two paths
Output = W₀x + scaled BAx. The bypass output is scaled by α/r (see the deep dive).
- Layer output h
Downstream layers see one combined output — the rest of the network cannot tell an adapter is there.
The consequences for operations are where LoRA earns its ubiquity. The trainable set is so small that gradients and optimizer state — the memory multiple that makes full fine-tuning multi-GPU work — nearly vanish. And the deliverable is not a model: it is an Adapter file containing only A and B for each adapted layer — megabytes-scale where a full model copy is gigabytes. That artifact shape unlocks a serving pattern: keep one copy of the base model in GPU memory and attach a different adapter per task, per customer, per tenant — swappable without redeploying the base. Or, if you only need one behavior, merge the update into the base (W₀ + BA becomes the new W) and serve a plain model with zero added inference latency — a property the paper makes explicit.
Bar chart comparing trainable parameter counts for one 4096 by 4096 weight matrix: full fine-tuning of the matrix trains 16,777,216 values, while LoRA trains 524,288 at rank 64, 131,072 at rank 16, and 65,536 at rank 8. The LoRA bars are tiny compared with the full bar, showing the parameter reduction.
The W + BA decomposition, with dimensions
Take one frozen weight matrix W₀ ∈ ℝ^(d×k) — d output dimensions, k input dimensions. Fine-tuning would learn an update ΔW of the same shape: d·k numbers. LoRA instead factorizes the update as ΔW = BA, with B ∈ ℝ^(d×r) and A ∈ ℝ^(r×k), where the rank r ≪ min(d, k). Trainable count: r·(d + k) instead of d·k. For d = k = 4096 and r = 8, that is 65,536 versus 16.8 million — the figure above.
The adapted forward pass is a sum of two paths: h = W₀x + (α/r)·BAx. The constant α is a scaling hyperparameter; dividing by r keeps the update’s magnitude roughly stable when you change the rank, so tuning r does not force retuning the learning rate. Initialization carries the elegance: A starts random Gaussian, B starts at zero, so BA = 0 and training begins exactly at the pretrained model — gradient descent then grows the update outward from a known-good start.
Why should a low-rank factorization suffice? A rank-r matrix can only express updates whose rows and columns live in an r-dimensional subspace — a severe constraint. The paper’s hypothesis, supported by its experiments, is that task adaptation has low intrinsic rank: the direction a pretrained model must move to acquire a narrow behavior is far simpler than its full parameter space. That is also LoRA’s honest limit — a behavior demanding a genuinely high-rank change (or knowledge the base never had) is beyond a small adapter, no matter the training data.
In the paper, adapters attach to the attention projection matrices (query and value); attaching to more matrices at lower rank is a common trade. At deployment, merging computes W = W₀ + (α/r)·BA once, offline — after which the adapter has zero runtime cost and the model is indistinguishable from a fully fine-tuned one in shape.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.