Precision reduction: the size-fidelity exchange rate
Quantization maps FP32 weights onto coarser grids: FP16/BF16 halve storage losslessly for most workloads; INT8 needs calibrated scales; INT4/FP4 make quality a real engineering conversation. Post-Training Quantization does all of this without retraining — the model is frozen, and scale factors absorb the compression.
PTQ Mechanics
Calibrate on a few hundred representative samples; per-channel scales = max|w| / grid_max. The rounding error budget per weight is half a grid step.
Weight-Only vs Weight-and-Activation
W4A16 keeps activations high precision and wins on quality; W8A8 unlocks integer Tensor Core paths and wins on speed. KV cache quantization is a third axis with its own trade.
Outlier Channels
A handful of activation channels carry magnitude 20-100× the median. Naive per-tensor scales waste the grid on them; per-channel or AWQ-style salience weighting rescues quality.
A 7B model at FP32 is 28 GB; INT4 is 3.5 GB — an 8× reduction with max per-weight error ≤ scale/2. Whether that error compounds into gibberish or disappears is what calibration decides.
- GPTQ quantizes layer-by-layer against calibration activations, minimizing output error rather than weight error.
- KV cache quantization (FP8) doubles usable context on the same silicon — often the highest-leverage rung for long-context serving.