Run Telemetry, Gradient Spikes, & Crash Recovery Engines
Pre-training across 20,000+ GPUs running continuously for 3 months suffers from inevitable hardware node crashes, cosmic ray bit flips, stragglers, and sudden loss spikes.
Monitoring Critical Telemetry Signals
Real-time dashboards track: 1. Loss curve & gradient norm ||g||. 2. Model FLOPs Utilization (MFU %). 3. Token throughput (tokens/sec/GPU). 4. HBM temperature & ECC memory errors.
Loss Spike Rewind Strategy
When loss spikes to infinity or NaN due to bad data shards or numerical explosion, automated recovery engines rewind to a checkpoint 2,000 steps prior, discard the bad data shard window, adjust LR, and resume.
Asynchronous Distributed Checkpointing
Saving a 70B model checkpoint (1.1 TB) every 1,000 steps uses background host RAM buffering and fast parallel NVMe pipelines to prevent blocking GPU computation.
Measuring the ratio of achieved theoretical matrix FLOPs against maximum hardware peak specs.
- Meta's Llama 3 405B training run sustained 38%+ MFU across 16,384 H100 GPUs while automatically recovering from over 400 node hardware failures without manual intervention.