Anatomy of a Single Training Step
Every pre-training run is an endless loop of a single fundamental iteration: Forward Pass -> Loss Calculation -> Backward Pass (Autodiff) -> Parameter Update.
1. The Forward Pass
Tokens pass through embedding layers, N Transformer blocks (RMSNorm, MHA/GQA, SwiGLU MLP), and an LM Head to output logits over vocabulary V. Activations must be cached in GPU HBM for backward pass calculation.
2. The Backward Pass (Backpropagation)
Applying the multivariable chain rule from the loss scalar backwards through layer outputs. Matrix multiplication transpose operations compute ∂L/∂W for each weight tensor.
3. The Optimizer Step
AdamW updates weight tensors using running exponentially weighted averages of gradients (m) and squared gradients (v), applying weight decay directly to weights.
Backward propagation translates output errors into weight updates by multiplying downstream error gradients by local activation transposes.
- In distributed clusters, backward pass execution is overlapped with gradient All-Reduce operations across the network fabric to mask communication latency.
- A step on a 70B model with 4M token global batch size takes ~2.5 seconds across 1,024 H100 GPUs.