Fusion, DCE and constant folding: paying the launch tax once
An eager transformer layer launches dozens of tiny kernels; each launch costs microseconds of CPU dispatch plus an HBM round-trip for intermediates. Graph optimization collapses pointwise chains into single kernels, deletes dead computation, and folds constants — before any hardware-specific tuning happens.
The Launch Tax
A kernel launch is ~5-10 µs of CPU-side overhead. A 32-layer model with 40 eager ops per layer burns milliseconds per step on dispatch alone at batch 1.
Fusion's Real Win
Bandwidth, not launches: a LayerNorm+GELU+add chain written eagerly writes each intermediate to HBM (7 GB/s of traffic per token at 7B). Fused, intermediates live in registers.
Compiler Passes
Dead-node elimination prunes unused branches; constant folding precomputes what weights alone determine; layout propagation picks NHWC/NCHW per target.
Fusing a 3-op chain over a 4k-token activation saves ~2 × 3 × activation-size of HBM traffic per layer, per step — often a 20-40% end-to-end decode speedup at batch 1.
- torch.compile's mode='max-autotune' and TensorRT both run these passes; the difference is schedule quality, not pass coverage.
- CUDA graphs capture the whole launch sequence once and replay it, eliminating dispatch jitter for decode loops.