Warps, SMs and Tensor Cores: mapping software onto H100
A CUDA thread is a fiction the scheduler resolves into 32-thread warps. Each of the H100's 132 SMs hosts up to 64 resident warps; when one stalls on memory, another issues. Tensor Cores are the matrix engines inside — fused multiply-accumulate tiles that give the GPU its headline FLOPs.
Latency Hiding by Occupancy
Warp schedulers swap in ready warps every cycle. Resident warp density is the currency: too few warps and every HBM stall becomes dead silicon.
Tensor Cores
Each SM packs 4 Tensor Cores executing 16×8×16 MMA tiles. Peak FLOPs live here; a kernel that doesn't feed them (via WMMA/MMA fragments) leaves 10× on the table.
Registers Are the Real Cache
Fused kernels keep intermediates in registers and shared memory; unfused chains pay HBM round-trips per op. This is why fusion matters more than kernel micro-tuning.
At 30% occupancy, ~5,000 warp slots idle. Latency bubbles appear exactly where resident density drops — the visual signature of an under-occupied kernel.
- FlashAttention's insight is scheduling, not math: tile attention so the softmax chain never leaves SRAM.
- Nsight Compute's 'achieved occupancy' vs 'theoretical occupancy' gap is the first thing kernel engineers check.