AI Engineering Visual Encyclopedia

ARCHITECTURE & FORMULATIONModule 5: Numerical Stability, Precision, & Acceleration

Mixed-Precision Math: FP16 vs. BF16 vs. FP8 & Dynamic Loss Scaling

Training in FP32 double/single precision is bottlenecked by HBM memory bandwidth and Tensor Core compute capability. Mixed-precision training runs GEMMs in 16-bit or 8-bit floats while keeping FP32 master weights for stability.

FP16 vs. BF16 Architecture

FP16 (5 exponent bits, 10 mantissa bits) has high precision but narrow dynamic range (max ~65,504), causing underflow/overflow gradients. BF16 (8 exponent bits, 7 mantissa bits) shares FP32's wide dynamic range, eliminating gradient underflow without loss scaling.

FP8 Pre-Training Era

FP8 (E4M3 for forward pass, E5M2 for backward pass) doubles Tensor Core TFLOPS on H100 GPUs while halving activation/weight memory, enabling massive compute acceleration.

Dynamic Loss Scaling (FP16)

Because FP16 underflows small gradients to zero, loss is multiplied by a scale factor S (e.g. 2^16) before backprop, then unscaled before optimizer updates.

MATHEMATICAL FORMULATION · DYNAMIC LOSS SCALING CORRECTION

Loss_scaled = Scale_S × Loss

Grad_master = ( 1 / Scale_S ) × Grad_FP16

Scaling keeps tiny floating point gradients within subnormal FP16 dynamic representation range.

REAL-WORLD PRODUCTION ENGINEERING

NVIDIA H100 Hopper SXM GPUs feature native FP8 Transformer Engines that dynamically switch between E4M3 and E5M2 per tensor.

{ "type": "object", "properties": { "query": { "type": "string", "minLength": 1, "maxLength": 300 }, "limit": { "type": "integer", "minimum": 1, "maximum": 10 } }, "required": [ "query" ], "additionalProperties": false }