ARCHITECTURE & FORMULATIONModule 3: SFT Deep Dive
Why low-rank adaptation works
Fine-tuning updates empirically live in low-dimensional subspaces: the change to a big weight matrix is close to low rank. LoRA exploits that by learning ΔW = B·A directly, freezing the original W and training only the small factors.
The Factorization
For a d×d weight matrix, full fine-tuning trains d² parameters; LoRA trains 2·r·d. With d=4096 and r=16, that is ~0.4% of the original.
Initialization
A starts Gaussian, B starts at zero — so ΔW = 0 at step one and the pretrained function is preserved exactly when training begins.
Scaling α
The update is scaled by α/r. Changing rank without re-tuning α silently changes the effective learning rate.
MATHEMATICAL FORMULATION · TRAINABLE PARAMETER SHARE
params_lora = target_frac · params_B · 10⁹ · (4·r·d) / d², frac = params_lora / (params_B × 10³M)
For 7B params with 50% of matrices targeted at r=16, d=4096: about 5.4M trainable parameters — under 0.1% of the model.
REAL-WORLD PRODUCTION ENGINEERING
- Target all attention projections plus MLP up/down projections for quality; attention-only LoRA underperforms on style and knowledge tasks.
- Adapter dropout and rank-stabilized scaling (rsLoRA) are common fixes when r grows past ~64.