The roofline: why decode lives on the memory wall
Every operation has an arithmetic intensity — FLOPs per byte moved. Above the knee, you're compute-bound; below it, you're streaming bytes and the bandwidth roof caps you. Inference's two phases sit on opposite sides, and almost every inference optimization is an attempt to move work up the intensity axis.
Von Neumann in Production
Data shuttles between memory and compute over a bus narrower than either side. SRAM (fast, tiny) vs HBM (big, slow) defines the trade every accelerator makes.
Decode's Math
One 7B FP16 token = 14 GB of weights, ~2×7 GFLOPs of compute. Intensity ≈ 1 FLOP/byte. Peak FLOPs are irrelevant; the bandwidth roof is destiny.
Escalation Ladders
Batching shares weight loads across requests; quantization shrinks the bytes; fusion keeps intermediates on-chip. All three are intensity upgrades.
For FP16 (b=2 bytes/param) decode AI ≈ 1. The H100 compute roof (989 TF) is 300× above its bandwidth roof at that intensity (3.35 TF). Utilization is not a bug — it is geometry.
- Batch size is the cheapest intensity knob: 64-way batching multiplies AI by ~64 without touching the model.
- NVLink-class interconnects exist because tensor parallelism multiplies bytes moved per FLOP even further.