SCENE 08 / 24 · TRANSFORMATION, COMPILATION & QUANTIZATION
Fuse the graph
Kernel fusion, dead-node elimination and constant folding shrink the launch tax.
KERNEL LAUNCHES · 4
Eager execution pays the launch tax per op and parks every intermediate tensor in HBM.
ARCHITECTURE & FORMULATIONModule 3: Model Transformation, Compilation & Quantization
Fusion, DCE and constant folding: paying the launch tax once
An eager transformer layer launches dozens of tiny kernels; each launch costs microseconds of CPU dispatch plus an HBM round-trip for intermediates. Graph optimization collapses pointwise chains into single kernels, deletes dead computation, and folds constants — before any hardware-specific tuning happens.
The Launch Tax
A kernel launch is ~5-10 µs of CPU-side overhead. A 32-layer model with 40 eager ops per layer burns milliseconds per step on dispatch alone at batch 1.
Fusion's Real Win
Bandwidth, not launches: a LayerNorm+GELU+add chain written eagerly writes each intermediate to HBM (7 GB/s of traffic per token at 7B). Fused, intermediates live in registers.
Compiler Passes
Dead-node elimination prunes unused branches; constant folding precomputes what weights alone determine; layout propagation picks NHWC/NCHW per target.
MATHEMATICAL FORMULATION · TRAFFIC SAVED BY FUSION
Fusing a 3-op chain over a 4k-token activation saves ~2 × 3 × activation-size of HBM traffic per layer, per step — often a 20-40% end-to-end decode speedup at batch 1.
REAL-WORLD PRODUCTION ENGINEERING
torch.compile's mode='max-autotune' and TensorRT both run these passes; the difference is schedule quality, not pass coverage.
CUDA graphs capture the whole launch sequence once and replay it, eliminating dispatch jitter for decode loops.