Learn and Practice the Building Blocks of DeepSeek-V4¶
A paper-aligned learning system for the mechanisms behind million-token context intelligence.
Readable PyTorch miniatures · executed notebooks · interactive walkthrough · measured experiments · tagged evidence, built from DeepSeek-V4 (arXiv:2606.19348v1). We demonstrate the mechanisms; we do not reproduce 1.6T parameters.
Start here¶
| if you want… | go to |
|---|---|
| the personal story | Publication essay: The Exchange Rates of a Million Tokens |
| the mental model in 15 minutes | Course 0 · The V4 Thesis |
| to see the token's journey | Interactive chapter walkthrough |
| to check a claim's provenance | Evidence Ledger |
The courses¶
| tab | course | paper coverage |
|---|---|---|
| 🏠 Home | this gateway | Abstract, Section 1 |
| ✍️ Journey | Publication essay | all sections, narrated |
| 🧱 Foundations | The V4 Thesis · Prerequisite library | Section 1 |
| 🏗️ Architecture | Overview (inherited designs) · CSA · HCA · mHC · Muon | Section 2 |
| 🎓 Training | Pre-training & long context · Post-training & OPD | Section 4, Section 5.1 |
| ⚙️ Systems | Infrastructures · Training framework · Post-training infra | Section 3, Section 5.2 |
| 📏 Evaluation | Reading results without fooling yourself | Section 4.3, Section 5.3–5.4 |
| 🎯 Conclusion | Limitations & future directions | Section 6 |
| 🔬 Practice | Executed notebook catalog | every mechanism |
The one-paragraph thesis¶
Million-token context makes memory movement, attention cost, KV geometry, communication, precision, and serving architecture inseparable. V4 answers with exchange rates: compression buys budget and spends recall; constraints buy depth-stability; bandwidth buys hidden time; training-time precision buys serving cost; scheduling discipline buys consolidation. This repo prices each rate at toy scale and tags what is PAPER, DERIVED, MEASURED, INFERRED, or UNKNOWN.
Why V4 looks like this¶
One constraint cascades through the entire architecture. Each step links to the lesson that prices it:
| causal step | priced in | |
|---|---|---|
| ● | 1M-token context becomes the product requirement | The V4 Thesis |
| ↓ | attention/KV cost becomes dominant | Cost curves · economics |
| ↓ | compress the memory representation | CSA |
| ↓ | CSA + HCA: two regimes, one job each | Hybrid attention |
| ↓ | new KV/cache geometry (lcm blocks, state, window) | KV cache system |
| ↓ | new inference & parallelism requirements | Inference framework |
| ↓ | MoE communication becomes critical | Expert parallelism |
| ↓ | overlap communication with computation | Infrastructures |
| ↓ | kernel design becomes architectural | Batch-invariant kernels |
| ↓ | precision becomes architectural: FP4 QAT | FP4 QAT |
| ↓ | post-training must preserve the efficiency regime | Post-training & OPD |
| ↓ | OPD + RL + agent infrastructure | OPD · Agent infra |
What changed from V3?¶
V4 is not a new model from nothing; it is a response to a new operating regime.
| component | DeepSeek-V3 | DeepSeek-V4 | why |
|---|---|---|---|
| MoE | inherited | refined: √Softplus affinities, sequence-wise balance loss, hash-routed first layers | scale |
| MTP | inherited | retained (depth 1) | training/inference utility |
| Attention | MLA lineage | CSA + HCA hybrid | million-token context |
| Residual stream | standard | mHC (Birkhoff-constrained lanes) | deep stability |
| Optimizer | Muon lineage | Muon with hybrid Newton-Schulz + RMS rescale | training efficiency |
| Precision | lower precision | FP4 quantization-aware training | serving/training economics |
| Systems | large-scale EP | tighter overlap/cache/kernel co-design | utilization |
Details and paper citations: Architecture overview.
The six exchange rates¶
Every mechanism on this site prices one of six trades. When a section teaches a mechanism, it is quoting one of these lines:
| rate | buys | spends | priced in |
|---|---|---|---|
| COMPRESSION | memory/compute | information/retrieval accuracy | CSA · HCA · hybrid |
| CONSTRAINT | stability guarantees | flexibility/representation freedom | mHC · Muon |
| BANDWIDTH | throughput/silence | engineering complexity | EP overlap · KV cache |
| LOW PRECISION | serving cost | numerical headroom | FP4 QAT |
| SCHEDULING | utilization/unbiased data | orchestration complexity | agent infra · training framework |
| DISTILLATION | consolidation of specialists | teacher/rollout cost | OPD |
Repository map¶
docs/ learning site (this content) + evidence ledger + plan
src/deepseek_v4_lab/
attention/ baseline GQA · CSA (dual-overlap + indexer) · HCA · hybrid
cache/ heterogeneous KV manager (lcm blocks, state cache)
moe/ √Softplus router, aux-loss-free balancing, hash routing
residual/ mHC: Sinkhorn-Knopp Birkhoff projection
mtp/ multi-token prediction head
optim/ Muon: hybrid Newton-Schulz
quant/ MXFP4 fake-quant + STE
dist/ EP overlap simulator
posttrain/ reverse-KL OPD machinery
agents/ WAL trajectory runner (crash-safe resume)
models/ tiny_v4: integrated chain (629K params) -> [The Whole Machine](whole-machine.md/)
experiments/ exp00–exp13, each answering one spec question
tests/ 65 tests: invariants, not smoke tests
artifacts/ config + metrics + summary per run
Quick start¶
pip install -e ".[dev]"
pytest # 65 passed (2026-08-21)
python experiments/exp02_csa.py # any experiment writes artifacts/
mkdocs serve # this learning site locally
Evidence discipline¶
Never turn "the paper reports" into "our experiment proves." Every claim carries PAPER / DERIVED / IMPLEMENTED / MEASURED / INFERRED / UNKNOWN; see the ledger. Our miniatures run at ~1/2,500,000th of Pro's scale; directions transfer; magnitudes do not.