Silent Data Corruption (SDC) & Transient Bit-Flips
Silent Data Corruption (SDC) refers to arithmetic or memory errors that occur without triggering hardware ECC interrupts or OS kernel panics. In massive clusters training trillion-parameter models, undetected bit-flips in matrix multiplications produce gradient explosions, loss spikes, and poisoned weights.
Root Causes of SDC
Sub-micron silicon defects, high-energy cosmic neutron strikes, subtle voltage fluctuations, and thermal degradation on aging tensor core circuits can produce incorrect arithmetic results (e.g., 1 + 1 = 3).
Loss Spikes & Weight Poisoning
An SDC event in a self-attention projection matrix can cause an activation value of 1e6 instead of 1.0. During backward pass, this propagates infinite/NaN gradients that permanently corrupt Adam optimizer momentum buffers.
Canary Matrix Multiplications & Checksums
Training frameworks inject periodic deterministically verifiable GEMM calculations (canaries) every 100 steps. If the output checksum differs from the expected value by even 1 LSB, the offending GPU is instantly isolated.
If single-GPU SDC rate lambda is 1 per 100,000 GPU-hours, a cluster of 16,384 GPUs operating for 30 days (11.8M GPU-hours) has a > 99.99% probability of experiencing multiple SDC events during the run.
- Implement gradient norm clipping and automated rollback: if gradient norm exceeds 10x moving average, discard the step and load the previous checkpoint.
- Run DCGM Diagnostic Level 3 (nvvs -t 3) including tensor core stress tests to identify degrading silicon before job assignment.
- Log full-precision checksums of model weights at each checkpoint to verify historical weight integrity.