Skip to content

Concept and Prerequisite Library

Attention fundamentals

  • Causal self-attention, each position attends only to earlier positions. Score \(s_{ij}=q_i\cdot k_j/\sqrt{d}\), softmax, weighted sum of values. Repo: attention/baseline.py, tested for strict causality (tests/test_attention.py).
  • KV cache, storing past keys/values turns \(O(n^2)\) training-style attention into \(O(n)\) decode-style. Cost: memory grows linearly with context. This linear growth is precisely what CSA/HCA attack.
  • GQA / MQA: fewer key/value heads than query heads (GQA) or one shared KV head (MQA). V4's core attention over compressed entries is MQA-shaped: one KV lane shared by all query heads [PAPER Section 2.3.1].

Mixture-of-Experts

  • Routing, per token, score all experts, activate top-k. Sparse activation = capacity without proportional compute.
  • Load balancing: unconstrained routers collapse onto favorites. V4 uses auxiliary-loss-free bias updates plus a small sequence-wise balance loss (weight 1e-4) [PAPER Section 2.1].

Optimization

  • Momentum / Nesterov: smoothed, look-ahead gradients. Muon feeds \(\mu M_t + G_t\) to its orthogonalizer [PAPER Alg 1 line 6].
  • Newton-Schulz iteration, polynomial matrix iteration driving singular values toward 1 without SVD. Fifth-order variant here: \(M_k = aM + b(MM^\top)M + c(MM^\top)^2M\).
  • ZeRO sharding: partition optimizer state across data-parallel ranks. Complication: Muon needs whole gradient matrices → knapsack bucketing [PAPER Section 3.4.1].

Information theory of compression

  • Max-entropy adversarial case, i.i.d. random tokens carry maximal information per token; compression must lose something. Real text has redundancy. All lab retrieval numbers are therefore lower bounds on production behavior [MEASURED framing].
  • Reverse KL: \(\mathrm{KL}(p\|q)=\sum p\log(p/q)\); mode-seeking (q concentrates where p does). OPD's consolidation force [PAPER Eq 29].
  • Straight-through estimator, forward through a non-differentiable op (quantize, top-k); backward as if identity. Used twice: FP4 QAT and our indexer STE phase.