Context Window Scaling & Quadratic Attention Memory
Standard Multi-Head Attention requires computing all pairs of interactions between query and key tokens, resulting in O(N^2) memory and compute complexity relative to sequence length N.
The O(N^2) Wall
At sequence length N=4,000, N^2 = 16M matrix elements. At N=128,000, N^2 = 16.38B matrix elements per attention head. Unoptimized attention score tensors instantly OOM even 80GB H100 GPUs.
KV Cache Expansion
During generation, Key and Value vectors for all past tokens must be preserved in VRAM. For a 128k context on Llama 3 70B, KV cache per user request consumes tens of GBs without Grouped-Query Attention.
Architectural Workarounds
Techniques like FlashAttention (tiled SRAM compute), RingAttention (sequence sharding across GPUs), and RoPE frequency scaling enable modern context lengths from 8k to 1M+ tokens.
Batch B, Heads H, Sequence N. Notice how doubling sequence length N quadruples required score buffer bytes.
- Context length expansion is typically done in two phases: pre-training on 8k tokens, then fine-tuning/extending on 128k+ tokens with adjusted RoPE theta.