Learn and Practice the Building Blocks of Kimi K3
A paper-aligned learning system for Kimi K3 (arXiv:2607.24653v2), combining explanatory miniatures, notebooks, visual lessons, and technical documentation. It teaches selected mechanisms and reported system results; it does not reproduce the 2.8T-parameter model or its production infrastructure.
Start learning
Begin with Course 0 — The K3 Thesis, then move through Architecture, Pretraining, Post-training, Infrastructure, Evaluation, and Case Studies. Each course uses the same contract:
Watch the computation → read the derivation → inspect retained notebook evidence → verify the claim boundary.
An animation explains; it does not measure. A local miniature establishes only the behavior it asserts. Full-scale K3 results remain paper-reported unless explicitly reproduced.
Choose a learning mode
| Mode | Use it for | Destination |
|---|---|---|
| Watch | Build a causal mental model and see state change. | Visual course gallery |
| Read | Follow derivations, diagrams, sources, and limitations. | Course 0 |
| Practice | Inspect retained outputs before choosing Colab, local execution, or download. | Notebook catalog |
| Look up | Resolve an unfamiliar prerequisite and follow its dependency chain. | Concept library |
| Verify | Trace a scene claim to its evidence class, paper location, and boundary. | Claim ledger |
| Read the source | Check the complete report rather than relying on this interpretation. | Kimi K3 paper |
Source: Author-created learning-mode guide for this Kimi K3 course; not a paper table.
Course route
| Course | Read | Watch | Practice |
|---|---|---|---|
| 0 — Thesis | Start here | Visual introduction | Evidence contract |
| 1 — Architecture | Derivations | Token journey | Notebooks 01–05 |
| 2 — Pretraining | Data to 1M | Training program | Notebook 08 |
| 3 — Post-training | Feedback system | Agentic learning | Notebook 06 |
| 4 — Infrastructure | Scale mechanics | Fleet walkthrough | Notebook 09 |
| 5A — Evaluation | Evidence literacy | Results without overclaiming | Notebook 07 |
| 5B — Cases | Trajectory dossiers | Cases and conclusion | Artifact/verification review |
Source: Author-created course route across Kimi K3 §§1–8, arXiv:2607.24653v2; not a paper table.
Before the infrastructure course, read Infrastructure Prerequisites if DP/ZeRO, PP/VP/1F1B, EP/CP, P2P Muon, DEP, OverlayBD, DSA, or GPU kernel toolchains are new to you.
Why This Exists
Scaling autoregressive language models to million-token contexts while maintaining high throughput and reasoning precision requires fundamental architectural innovations beyond traditional \(O(N^2)\) Transformers:
- State Retention Efficiency: KV-cache storage grows with sequence length, while Kimi Delta Attention (KDA) retains a fixed-size state matrix per head. Its readable reference update is \(\bar S_t=\operatorname{Diag}(\alpha_t)S_{t-1}\), \(e_t=v_t-\bar S_t^Tk_t\), and \(S_t=\bar S_t+\beta_tk_te_t^T\).
- Low-Rank Compression: Gated MLA compresses Key/Value representations into a low-rank latent vector \(c_t = W_c x_t\), dropping Rotary Position Encodings (NoPE) as KDA layers supply positional recurrence.
- Depth Routing: Block Attention Residuals (AttnRes) replaces standard linear residual additions with softmax depth attention weighting across model layers.
- Numerical MoE Stability: Stable LatentMoE introduces SiTU-GLU bounded activation functions and dynamic Quantile Load Balancing without auxiliary loss penalties.
High-Level Architecture
flowchart LR
In[Input Tokens / Vision Patches] --> Patch[MoonViT-V2 Spatial Patchify]
Patch --> KDA[KDA Layers 1-3: recurrent state]
KDA --> MLA[Gated MLA Layer: Low-Rank NoPE]
MLA --> AttnRes[Block Attention Residuals: Depth Routing]
AttnRes --> MoE[Stable LatentMoE: 896 Experts / Quantile Balancer]
MoE --> Head[Per-Head Muon Optimized Outputs]Source: Author-created overview of Kimi K3 Fig. 2 and §§2.1–2.5; not a paper figure.
Paper Section & Building Block Mapping
| Section & Topic | Core Mathematical Formula | PyTorch Implementation | Interactive Visualizer |
|---|---|---|---|
| 2.1.1 KDA Attention | \(\bar S=\operatorname{Diag}(\alpha)S\), \(e=v-\bar S^Tk\), \(S=\bar S+\beta ke^T\) | src/architecture/kda.py |
KDA State-Handoff Lesson |
| 2.1.2 Gated MLA (NoPE) | \(c_t = W_c x_t, \quad y_t = W_o[\operatorname{Sigmoid}(W_g x_t) \odot \text{MLA}(x_t)]\) | src/architecture/gated_mla.py |
Interactive MLA NoPE Explainer |
| 2.2 Attention Residuals | \(\alpha_{i \to l} = \operatorname{softmax}(w_l \cdot \text{RMSNorm}(b_i) / \sqrt{d})\) | src/architecture/attn_res.py |
AttnRes Depth Routing |
| 2.3 Stable LatentMoE | \(\operatorname{SiTU-GLU}(x) = (\beta_1 \tanh(x_g/\beta_1) \odot \sigma(x_g)) \odot (\beta_2 \tanh(x_u/\beta_2))\) | src/architecture/stable_latent_moe.py |
Quantile Load Balancer |
| 2.5 Muon Optimizer | \(X_{k+1} = a X_k + b X_k X_k^\top X_k + c X_k (X_k^\top X_k)^2\) | src/training/optimizer.py |
Muon Convergence Simulator |
Source: Author-created repository map aligned to Kimi K3 §§2.1–2.5, arXiv:2607.24653v2; formulas are teaching summaries and links point to local miniatures.