Mixed-Precision Math: FP16 vs. BF16 vs. FP8 & Dynamic Loss Scaling
Training in FP32 double/single precision is bottlenecked by HBM memory bandwidth and Tensor Core compute capability. Mixed-precision training runs GEMMs in 16-bit or 8-bit floats while keeping FP32 master weights for stability.
FP16 vs. BF16 Architecture
FP16 (5 exponent bits, 10 mantissa bits) has high precision but narrow dynamic range (max ~65,504), causing underflow/overflow gradients. BF16 (8 exponent bits, 7 mantissa bits) shares FP32's wide dynamic range, eliminating gradient underflow without loss scaling.
FP8 Pre-Training Era
FP8 (E4M3 for forward pass, E5M2 for backward pass) doubles Tensor Core TFLOPS on H100 GPUs while halving activation/weight memory, enabling massive compute acceleration.
Dynamic Loss Scaling (FP16)
Because FP16 underflows small gradients to zero, loss is multiplied by a scale factor S (e.g. 2^16) before backprop, then unscaled before optimizer updates.
Scaling keeps tiny floating point gradients within subnormal FP16 dynamic representation range.
- NVIDIA H100 Hopper SXM GPUs feature native FP8 Transformer Engines that dynamically switch between E4M3 and E5M2 per tensor.