HBM3e Bandwidth & Roofline Knee
High-Bandwidth Memory (HBM3e) utilizes 3D-stacked DRAM dies connected to the GPU processor via a silicon interposer with thousands of microscopic microbumps. The Roofline Model identifies the exact arithmetic intensity (FLOPs/Byte) required to transition an operation from memory-bandwidth-bound to compute-bound.
Silicon Interposer & Through-Silicon Vias (TSVs)
Unlike standard GDDR6 chips connected across motherboard traces, HBM stacks sit on the same substrate silicon interposer as the GPU die, providing 1024-bit wide buses per stack with massive aggregate memory bandwidth (up to 3.35 TB/s on H200, 8.0 TB/s on B200).
The Roofline Knee & Arithmetic Intensity
Arithmetic intensity is the ratio of operations performed to bytes transferred from memory (FLOP/Byte). The knee point = Peak Compute / Peak Memory Bandwidth. For an H100 with 1,000 TFLOPs FP16 compute and 3.35 TB/s HBM bandwidth, the knee is ~298 FLOPs/Byte.
Memory-Bound vs Compute-Bound Kernels
Element-wise activations (SiLU, GELU, LayerNorm) and un-fused attention have low arithmetic intensity (< 10 FLOP/Byte) and waste 95% of GPU tensor cores waiting on HBM. Large GEMMs (matrix multiplications) have high intensity (> 500 FLOP/Byte) and fully saturate compute.
When operational intensity I < (P_peak / BW_mem), attainable compute scales linearly with memory bandwidth. Only when I exceeds the roofline knee does performance reach the horizontal ceiling of peak tensor core throughput.
- FlashAttention-2/3 overcomes the memory wall by tiling Q, K, V blocks into high-speed SRAM (18-33 TB/s), avoiding intermediate HBM read/writes for the N x N attention matrix.
- Triton and CUDA kernel fusion combine consecutive element-wise operations into a single kernel launch to maximize data reuse in L1/L2 cache.
- Monitor SM occupancy vs DRAM bandwidth utilization using Nsight Compute to pinpoint memory-bound bottlenecks.