Model FLOPs Utilization (MFU) vs MBU Accounting
Hardware FLOPS ratings provided on spec sheets represent theoretical peak limits under ideal conditions. Model FLOPs Utilization (MFU) and Model Bandwidth Utilization (MBU) are the gold-standard metrics that measure true hardware efficiency during actual distributed execution.
Model FLOPs Utilization (MFU)
MFU measures the ratio of theoretical matrix operations required by the transformer architecture (6 * Params * Tokens per step) to the peak theoretical hardware FLOPS of the cluster. World-class pretraining runs achieve 45%–55% MFU.
Hardware FLOPs Utilization (HFU)
HFU includes recomputation (activation checkpointing) FLOPs. While HFU may be 65%, true MFU measures only productive forward/backward work. Comparing MFU to HFU exposes recomputation overhead.
Model Bandwidth Utilization (MBU)
During autoregressive inference token decoding, the system is memory-bound. MBU measures the fraction of peak HBM memory bandwidth achieved while loading model weights and KV cache per token.
For a 70B parameter model training at 3,200 tokens/sec across 64 H100 GPUs (peak FP16 = 989 TFLOPs each): MFU = (3200 * 6 * 70e9) / (64 * 989e12) = 1.344e15 / 6.329e16 = 21.2%. Optimizing kernels to reach 50% MFU doubles throughput to 7,500 tokens/sec without adding hardware.
- Measure MFU at every scale step during cluster bring-up; a sudden drop in MFU when scaling from 64 to 512 GPUs indicates network communication bottlenecks.
- Profile pipeline bubbles in 1F1B schedules: bubble fraction F_bubble = (p - 1) / (p - 1 + m), where p is pipeline depth and m is number of microbatches.
- Incorporate FlashAttention, fused SwiGLU, and FP8 precision to maximize arithmetic intensity and MFU.