Final Evaluation: the dependency chain, executed¶
Spec Section 17 defines success as: we can explain, implement, instrument, and experimentally interrogate the mechanisms that make DeepSeek-V4's million-token architecture plausible. This page maps every link of the chain to what we built and measured.
| chain link | artifact | status |
|---|---|---|
| Long-context requirement | Exp01 baseline curves (quadratic FLOPs / linear KV) | MEASURED+DERIVED |
| Hybrid compressed attention | attention/csa.py, hca.py, Exps 02–04 |
IMPLEMENTED+MEASURED |
| KV representation | compression ratios: CSA ≈3% of vanilla cache [DERIVED] | test-validated |
| Cache + parallelism | heterogeneous manager, lcm blocks, persist/reuse invariants (Exp06); overlap simulator crossover law (Exp08) | IMPLEMENTED+MEASURED |
| MoE capacity | Sqrt(Softplus) routing, bias balancing, utilization tracking | IMPLEMENTED |
| Communication overlap | speedup saturation at bandwidth crossover | MEASURED (simulator) |
| Kernel design principles | determinism/batch-invariance lessons surfaced by bugs (update_bias during eval broke bitwise equality) |
MEASURED (accidentally, then deliberately) |
| Precision | MXFP4 STE, PTQ-vs-QAT with finetune control (Exp10) | MEASURED |
| Training stability | mHC Birkhoff closure tests; Muon hybrid NS ±0.15 of σ=1 (Exp07/09) | IMPLEMENTED+TESTED |
| Post-training | OPD machinery + scheduling-collapse finding + toy-scale negative result (Exp11) | MEASURED (incl. negative) |
| Agent execution | WAL-resume runner, byte-identical crash recovery, fast-forward replay (Exp12) | MEASURED |
Evidence tags: PAPER stated by DeepSeek · DERIVED mathematically implied · IMPLEMENTED reproduced in this project · MEASURED observed in our experiments · INFERRED our interpretation · UNKNOWN not established. Full definitions: evidence ledger
Integrated model (Exp13)¶
models/tiny_v4.py, 629K params, all seven mechanisms in one trainable
stack (hybrid CSA/HCA in mHC lanes, MoE FFN, MTP head, Muon/AdamW split).
- End-to-end training through TRUE top-k sparse attention succeeds: sparse-phase retrieval loss → 0.003–0.11 [MEASURED]
- Sparse eval accuracy 28.9% (chance 6.25%; dedicated 2-layer CSA from Exp02: ~88%): the added stack complexity (MoE + mHC + HCA alternation) makes the lightning indexer's job harder at this budget [MEASURED, limitation stated rather than tuned away]
What the paper proves vs what we showed¶
Everything PAPER-tagged in docs/evidence-ledger.md belongs to
arXiv:2606.19348v1. Everything MEASURED here belongs to toy-scale runs on a
CPU laptop with 3.7GB RAM, single seeds, synthetic max-entropy data. The
directions agree; magnitudes do not, and were never expected to.