ARCHITECTURE & FORMULATIONModule 3: Modern Architectural Specifications (The Pre-Train Recipe)
Rotary Position Embeddings (RoPE)
RoPE (Su et al.) encodes relative positional information by multiplying 2D sub-vectors of Query and Key representations by rotation matrices proportional to sequence position.
Relative via Absolute Rotation
Instead of adding position embeddings to input tokens, RoPE rotates Q and K vectors in complex 2D planes. The dot product <R_m Q, R_n K> depends purely on relative distance (m - n).
No Extra Memory/Parameters
RoPE requires zero additional model parameters and is applied on-the-fly inside attention heads.
Length Extrapolation (RoPE Scaling)
By scaling the base theta frequency (e.g. from 10,000 to 500,000 or YaRN/linear scaling), models trained at 8k context can extrapolate to 128k+ tokens cleanly.
MATHEMATICAL FORMULATION · 2D ROPE ROTATION OPERATOR
R_(θ, m) = [ cos(m θ_i) -sin(m θ_i) ]
[ sin(m θ_i) cos(m θ_i) ]
Rotates vector pairs at position m using dimension-dependent frequency θ_i = 10000^(-2(i-1)/d).
REAL-WORLD PRODUCTION ENGINEERING
- Every top open-weights foundation LLM (Llama 3, Qwen 2.5, Mistral, Gemma 2) uses RoPE position embeddings.