Daly's Formula: Optimal Checkpoint Frequency
In large-scale AI training with thousands of GPUs, hardware components inevitably fail. Checkpointing saves model weights and optimizer states to persistent storage. Daly's (and Young's) formula determines the exact mathematical checkpoint interval that minimizes lost compute time.
The Checkpointing Trade-Off
Checkpointing too frequently incurs heavy I/O blocking overhead (delta). Checkpointing too infrequently results in catastrophic lost compute (rework time) when a node crashes and the run must roll back to the last save point.
Stochastic Failure Probability (Poisson Process)
Cluster failures follow a Poisson distribution with Mean Time Between Failures (MTBF). If single-GPU MTBF is 3 years, a cluster of 16,384 GPUs experiences a hardware failure approximately every 1.6 hours.
Tiered Asynchronous Checkpointing
State is quickly snapshotted to local node NVMe flash in 2–5 seconds (blocking), then uploaded to cloud object storage (S3/GCS) in the background asynchronously without pausing GPU training.
For a cluster with MTBF = 10 hours (36,000s) and checkpoint dump time delta = 180s (3 min), optimal interval tau_opt = sqrt(2 * 180 * 36000) - 180 = 3600 - 180 = 3420 seconds (~57 minutes).
- Implement PyTorch Distributed Checkpoint (torch.distributed.checkpoint) for non-blocking asynchronous state saving.
- Track cluster MTBF continuously in monitoring dashboards and dynamically adjust checkpoint frequency if failure rates spike.
- Use distributed deduplication: save optimizer states in sharded FP32 slices while saving model weights in BF16.