POST-TRAINING & DISTILLATION

MODULE 3: SFT DEEP DIVE · SCENE 08

QLoRA Memory: 4-bit Base, Full-Pipeline Bills

NF4 quantization constants, PagedAdamW, and the VRAM ledger that fits a 7B tune on a single consumer GPU.

MODEL SIZE7B PARAMSFULL · 112 GBQLORA · 4.7 GB · FITS 24GB ✓
NF4 BASE · 3.9 GBBF16 ADAPTERS + PAGED ADAMW · 0.9 GB24× REDUCTION VS FULL FINE-TUNE (112 GB AT 16 B/PARAM)

Everything big is frozen and quantized; everything trainable is tiny. Paged optimizers spill AdamW moments to CPU RAM on activation spikes.

ARCHITECTURE & FORMULATIONModule 3: SFT Deep Dive

QLoRA memory layout, byte by byte

QLoRA's contribution is an accounting trick plus two engineering details: NF4 quantization matched to the weight distribution, and paged optimizers that spill optimizer state to CPU RAM on memory spikes.

NF4 + Double Quantization

NormalFloat4 matches the empirical distribution of trained weights better than uniform int4. Quantizing the quantization constants saves ~0.4 bits/param.

Paged Optimizers

NVIDIA unified memory lets AdamW's moments page between GPU and CPU RAM, absorbing the activation-memory spikes of long sequences.

The Ledger

4-bit base + bf16 adapters + fp32 optimizer moments for adapters only. Everything big is frozen; everything trainable is tiny.

MATHEMATICAL FORMULATION · FULL FINE-TUNE VS QLORA
full_GB = params_B × 16 vs qlora_GB ≈ params_B × 0.55 + adapters

7B model: full fine-tune ≈ 112 GB (bf16 weights, bf16 grads, fp32 Adam moments and master copy). QLoRA ≈ 3.9 GB base + <2 GB adapters — a 20× reduction.

REAL-WORLD PRODUCTION ENGINEERING
  • Gradient checkpointing plus QLoRA is what makes 13B-class fine-tunes fit on 16–24GB cards.
  • Beware merged-precision loss: merging bf16 adapters into an NF4 base, then dequantizing, can lose a little quality — merge into a bf16 copy for deployment.
← 07 · LoRA: Why Low-Rank Adaptation Works
09 · Catastrophic Forgetting & Domain Adaptation →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]