Causal Attention Masking & Temporal Flow
To enforce left-to-right temporal order during parallel training, self-attention score matrices are masked with negative infinity above the main diagonal before applying Softmax.
Lower-Triangular Masking
Position i is prohibited from attending to position j if j > i. Setting S_(i,j) = -∞ forces Softmax(S)_(i,j) = 0, zeroing out attention weight.
Parallel Training Efficiency
Without causal masking, training on sequence length N would require N sequential forward passes. Causal masking allows all N token positions to be trained concurrently in a single forward/backward pass.
Waste Elimination
Exactly (N-1)/(2N) ≈ 50% of the attention matrix represents upper-triangular masked invalid comparisons. FlashAttention and FlashDecoding exploit this geometry to skip computing 50% of tile scores entirely.
Adding -∞ in the mask matrix M sets exponentiated values to zero in the Softmax row normalization.
- Modern CUDA kernels for causal attention use triangular loop bounds in GPU thread blocks, saving 50% FLOPs and DRAM bandwidth.