Course 1: Architecture¶
Paper coverage: Section 2, Section 2.1–Section 2.4 Interactive: chapter walkthrough · Code:
src/deepseek_v4_lab/attention/,residual/,moe/
What V4 kept from DeepSeek-V3 [Section 2.1]¶
V4 is deliberately conservative. Retained unchanged: the Transformer skeleton, Multi-Token Prediction (depth 1), the tokenizer (128K vocab), token-splitting and Fill-in-Middle, auxiliary-loss-free MoE balancing (plus a new sequence-wise balance loss, weight 1e-4). Changed at the margins:
- routing affinity \(\operatorname{Sigmoid}\to\sqrt{\operatorname{Softplus}(\cdot)}\), smoother positive scores;
- hash routing for the first 3 MoE layers, target experts fixed by token ID; the paper gives no reason [UNKNOWN];
- the constraint on routing-target node count was removed, with parallelism redesigned around it.
The point of the table on the Home gateway: V4 is not a new model from nothing; it is a response to a new operating regime. Most of the design is inherited V3 machinery; what changed changed because million-token context moved the constraints.
The three upgrades¶
| upgrade | section | one-line exchange rate |
|---|---|---|
| Hybrid CSA/HCA attention | Section 2.3 | memory budget ↔ retrieval precision |
| mHC residual streams | Section 2.2 | unconstrained expressivity ↔ bounded depth propagation |
| Muon optimizer | Section 2.4 | element-wise simplicity ↔ spectral control |
Model configurations [PAPER Section 4.2.1]¶
| Pro | Flash | |
|---|---|---|
| layers / d_model | 61 / 7168 | 43 / 4096 |
| first layers | 2× HCA | 2× pure SWA |
| CSA: m / top-k / indexer heads / c_I | 4 / 1024 / 64 / 128 | 4 / 512 / 64 / 128 |
| HCA: m′ | 128 | 128 |
| query heads / head dim c / d_c | 128 / 512 / 1536 | 64 / 512 / 1024 |
| out-proj groups g / d_g | 16 / 1024 | 8 / 1024 |
| sliding window n_win | 128 | 128 |
| MoE: shared + routed / top-k / expert dim | 1+384 / 6 / 3072 | 1+256 / 6 / 2048 |
| mHC n_hc / Sinkhorn iters | 4 / 20 | 4 / 20 |
Note the ratios worth internalizing: compression rates differ by 32× between the two attention regimes (4 vs 128); routed-expert count differs by ~10× while activated compute stays fixed by top-6.
Chapter pages¶
- CSA, Compressed Sparse Attention, overlapped pooling + lightning indexer + MQA core.
- HCA, Heavily Compressed Attention, extreme pooling, dense readout.
- mHC, Manifold-Constrained Hyper-Connections, Birkhoff-constrained lane mixing.
- Muon Optimizer, hybrid Newton-Schulz orthogonalization.
Lab checkpoints for this course¶
exp02, indexer recall vs k after staged curriculum [MEASURED]exp03, HCA accuracy cliff past m′≈8 on adversarial data [MEASURED]exp04, four-way frontier table; hybrid ≈ CSA accuracy at half KV cost [MEASURED]exp07, σ(B)≤1 and closure under multiplication to depth 128 [IMPLEMENTED]