AI Engineering Visual Encyclopedia

ARCHITECTURE & FORMULATIONModule 1: Objectives & Paradigm

Causal Attention Masking & Temporal Flow

To enforce left-to-right temporal order during parallel training, self-attention score matrices are masked with negative infinity above the main diagonal before applying Softmax.

Lower-Triangular Masking

Position i is prohibited from attending to position j if j > i. Setting S_(i,j) = -∞ forces Softmax(S)_(i,j) = 0, zeroing out attention weight.

Parallel Training Efficiency

Without causal masking, training on sequence length N would require N sequential forward passes. Causal masking allows all N token positions to be trained concurrently in a single forward/backward pass.

Waste Elimination

Exactly (N-1)/(2N) ≈ 50% of the attention matrix represents upper-triangular masked invalid comparisons. FlashAttention and FlashDecoding exploit this geometry to skip computing 50% of tile scores entirely.

MATHEMATICAL FORMULATION · MASKED ATTENTION SOFTMAX

Attention(Q, K, V) = Softmax( (Q Kᵀ / √d_k) + M ) V

where M_(i,j) = 0 if i ≥ j, else -∞

Adding -∞ in the mask matrix M sets exponentiated values to zero in the Softmax row normalization.

REAL-WORLD PRODUCTION ENGINEERING

Modern CUDA kernels for causal attention use triangular loop bounds in GPU thread blocks, saving 50% FLOPs and DRAM bandwidth.

{ "type": "object", "properties": { "query": { "type": "string", "minLength": 1, "maxLength": 300 }, "limit": { "type": "integer", "minimum": 1, "maximum": 10 } }, "required": [ "query" ], "additionalProperties": false }

The mask that lets time flow one way

Causal Attention Masking & Temporal Flow

Lower-Triangular Masking

Parallel Training Efficiency

Waste Elimination