AI ENGVisual Encyclopedia

MODULE 3: SFT DEEP DIVE · SCENE 08

QLoRA Memory: 4-bit Base, Full-Pipeline Bills

NF4 quantization constants, PagedAdamW, and the VRAM ledger that fits a 7B tune on a single consumer GPU.

MODEL SIZE7B PARAMSFULL · 112 GBQLORA · 4.7 GB · FITS 24GB ✓
NF4 BASE · 3.9 GBBF16 ADAPTERS + PAGED ADAMW · 0.9 GB24× REDUCTION VS FULL FINE-TUNE (112 GB AT 16 B/PARAM)

Everything big is frozen and quantized; everything trainable is tiny. Paged optimizers spill AdamW moments to CPU RAM on activation spikes.

TECHNICAL BREAKDOWNModule 3: SFT Deep Dive

QLoRA memory layout, byte by byte

QLoRA's contribution is an accounting trick plus two engineering details: NF4 quantization matched to the weight distribution, and paged optimizers that spill optimizer state to CPU RAM on memory spikes.

NF4 + Double Quantization

NormalFloat4 matches the empirical distribution of trained weights better than uniform int4. Quantizing the quantization constants saves ~0.4 bits/param.

Paged Optimizers

NVIDIA unified memory lets AdamW's moments page between GPU and CPU RAM, absorbing the activation-memory spikes of long sequences.

The Ledger

4-bit base + bf16 adapters + fp32 optimizer moments for adapters only. Everything big is frozen; everything trainable is tiny.

MATHEMATICAL FORMULATION · FULL FINE-TUNE VS QLORA
full_GB = params_B × 16 vs qlora_GB ≈ params_B × 0.55 + adapters

7B model: full fine-tune ≈ 112 GB (bf16 weights, bf16 grads, fp32 Adam moments and master copy). QLoRA ≈ 3.9 GB base + <2 GB adapters — a 20× reduction.

REAL-WORLD PRODUCTION ENGINEERING
  • Gradient checkpointing plus QLoRA is what makes 13B-class fine-tunes fit on 16–24GB cards.
  • Beware merged-precision loss: merging bf16 adapters into an NF4 base, then dequantizing, can lose a little quality — merge into a bf16 copy for deployment.
← 07 · LoRA: Why Low-Rank Adaptation Works
09 · Catastrophic Forgetting & Domain Adaptation →