LoRA & QLoRA: the parameter-efficient memory trade
Instead of updating every weight, LoRA freezes the base model and learns tiny low-rank matrices whose product approximates the needed weight change. QLoRA stacks 4-bit quantization of the frozen base on top, turning a multi-GPU job into a single-GPU job.
Low-Rank Adapters
The weight update ΔW is factorized as B·A where A is r×d and B is d×r. With r ≪ d, trainable parameters collapse to a fraction of a percent.
4-bit Base
QLoRA stores the frozen base in NF4 (~0.5 bytes/param) with double quantization and paged optimizers, while LoRA adapters stay in bf16.
Merge at Deploy
After training, W' = W + B·A merges back into the base weights, so inference cost is identical to the original model — zero latency overhead.
A 7B model needs ~3.9 GB for the 4-bit base plus a few GB of adapters and optimizer state; a full fine-tune of the same model needs ~7 × 16 = 112 GB.
- r=8–64 on q_proj/k_proj/v_proj/o_proj + MLP is the practical sweet spot; higher rank helps style-heavy domains more than factual recall.
- QLoRA made it routine to fine-tune a 33B model on a single 24GB consumer GPU — the democratization moment for instruction tuning.