The Exchange Rates of a Million Tokens¶
What rebuilding DeepSeek-V4 at 1/2,500,000th scale taught me about what each mechanism buys, and what it costs.
Date: August 21, 2026 | Author: Tanvir | Category: Frontier AI Architecture & Learning Systems
TL;DR: I spent weeks rebuilding the mechanisms behind DeepSeek-V4 (arXiv:2606.19348v1) as small, inspectable PyTorch, 65 tests, 13 experiments, a 629K-parameter integrated stack against a 1.6T-parameter original. The lesson that emerged was not architectural but economic: every mechanism in this model is an exchange rate. Memory buys certainty. Compression buys budget and spends recall. Bandwidth buys silence. Precision, bought during training, pays back at serving. Consolidation has a scheduling price. And determinism, the rate nobody prices, turns out to underwrite everything else. This essay is the ledger.
1 · The sticker price was never the story¶
The DeepSeek-V4 report opens with numbers built to stop the scroll: 1.6 trillion total parameters (49B active) for V4-Pro, 284B (13B active) for V4-Flash, both natively serving one-million-token contexts, trained on more than 32T tokens.
Then comes the number I find genuinely interesting: at one million tokens of context, V4-Pro needs only 27% of V3.2's single-token FLOPs and 10% of its KV cache.
That is not a capability claim. It is a price list. It says: we found a cheaper way to remember.
So instead of asking "how does DeepSeek-V4 work," my project asked a question I could actually answer on a laptop:
What does a million tokens cost under each memory architecture, and what does every mechanism buy with that bill?
Everything below is measured in deepseek-v4-lab, a 1/2,500,000th-scale reconstruction: same equations, same invariants, toy dimensions, single seeds, honest tags separating what the paper reports from what I executed.
2 · Rate #1: memory buys certainty¶
Vanilla causal attention is the luxury option. Every token can address every earlier token exactly. The invoice:
KV cache ∝ n × d (linear, but relentless)
attention ∝ n² (quadratic, and it wins eventually)
My baseline experiment (exp01) confirmed the curve shape empirically up to 4K tokens on CPU and analytically to 1M: at GQA-like toy dimensions, a single layer owes roughly 1 GB of cache per million-token context. Multiply by layers and you understand, viscerally, why the paper's first move is not "a better attention" but "a cheaper representation of memory."
The first exchange rate is therefore trivial and brutal: exactness costs quadratically.
3 · Rate #2: bytes buy budget, and spend recall¶
Compressed Sparse Attention is V4's opening bid. Each KV entry pools m = 4 tokens, and not by naive averaging. Two parallel series (Cᵃ, Cᵇ) overlap across block boundaries so that consecutive entries share their boundary tokens under one joint softmax, then a lightning indexer scores compressed blocks with cheap ReLU-gated dot products and selects top-k.
Rebuilding this (exp02) produced the finding I did not expect:
You cannot train the indexer densely. With dense-over-compressed warmup only, true top-k evaluation collapsed to near chance. The indexer needs its own sparse fine-tuning phase with straight-through gradients, which is precisely why the paper introduces sparsity as a staged curriculum (dense warmup → indexer warmup → sparse training) rather than flipping a switch.
With that curriculum, needle-block recall climbed from ~2% to ~72–89%, and the cache shrank to 2.9% of the vanilla baseline. That is the second exchange rate, stated honestly: compression converts a memory problem into a retrieval-quality problem. The bytes are not free; they are financed.
4 · Why two currencies beat one¶
Heavily Compressed Attention (HCA) pushes pooling to m′ = 128 with no sparse selection at all, dense attention over heavily pooled entries. My sweep (exp03) hit a cliff: past m′ ≈ 8, accuracy fell to chance on max-entropy data.
Which raises the paper's actual design question: why carry both regimes interleaved?
My four-way comparison (exp04) gave the toy-scale answer:
| variant | retrieval acc | KV @1M ctx |
|---|---|---|
| vanilla | 0.88 | 1024 MB |
| CSA (m=4, k=8) | 0.24 | 48 MB |
| HCA (m′=32) | 0.06 | 4 MB |
| hybrid (interleaved) | 0.21 | 26 MB |
The hybrid matches most of CSA's retrieval at roughly half its memory; each regime covers the other's blind spot. One abstraction cannot satisfy incompatible requirements; two can trade places depending on the layer.
5 · Rate #3: freedom buys instability; constraints buy depth¶
Hyper-connections widen the residual stream into lanes that mix learnedly between layers. Unconstrained, that mixing matrix B can stretch signal exponentially with depth. V4 constrains B to the Birkhoff polytope, doubly stochastic matrices, projected via Sinkhorn-Knopp iterations.
My study (exp07) verified the two properties that make the constraint worth its price: σ(B) ≤ 1 (non-expansive) and closure under multiplication, so a 128-layer stack inherits boundedness by construction rather than by luck. On gentle initialization both constrained and unconstrained variants are stable; the constraint's value is the guarantee, not the default.
Muon, the optimizer, makes the same trade in a different currency: it abandons element-wise simplicity (AdamW-style) for orthogonalized updates via hybrid Newton-Schulz iterations, eight aggressive coefficient sets, then two settling ones. Rebuilding it surfaced a caveat worth publishing: in bf16, rounding seeds exact-zero singular directions with noise, and Newton-Schulz drives every direction toward σ = 1. Production never sees this because real gradients are dense, but it explains why these implementations normalize by Frobenius norm and quietly depend on gradient density.
6 · Rate #4: bandwidth buys silence¶
Expert parallelism has a communication tax: dispatch tokens out, combine results back. The paper's Section 3.1 observation is that per MoE layer, communication time is less than computation time, so if fused into one pipelined wave schedule, compute stays the bottleneck and bandwidth stops mattering.
The condition is beautiful: hiding is complete when
C / B ≤ V_comp / V_comm
For V4-Pro's SwiGLU experts that simplifies to 2d = 6144 FLOPs per Byte: each GB/s of interconnect shields ~6 TFLOP/s of compute from ever waiting. My simulator (exp08) reproduced the law structurally: speedup saturates right where the ratio crosses over, and instantiated the same arithmetic at Flash-scale dimensions (~4096 FLOPs/Byte).
Silence, it turns out, has a listed price in hardware datasheets.
7 · Rate #5: precision, bought during training, pays at serving¶
MXFP4 quantization of expert weights and the indexer's QK path is where V4 most explicitly blurs "architecture" and "training." Quantize a trained model after the fact (PTQ) and you degrade it. Quantize during training with a straight-through estimator and the weights migrate onto the FP4 grid themselves.
My controlled comparison (exp10) made the distinction measurable:
| variant | eval loss |
|---|---|
| fp32 | 2.143 |
| PTQ-FP4 | 2.145 |
| fp32 + same extra finetune steps (control) | 2.117 |
| QAT-FP4 | 2.118 |
QAT-FP4 statistically matches the control that kept full precision through the same extra training. That gap between the first and last row is the meaning of "quantization-aware": precision became part of the objective. The same philosophy appears earlier in the pipeline: sparsity itself is introduced gradually, as a curriculum.
8 · Rate #6: consolidation has a scheduling price¶
V4's post-training replaces mixed RL entirely: train domain specialists with SFT + GRPO, then consolidate >10 of them via on-policy distillation, reverse KL against teachers, scored on trajectories the student generated itself.
My toy version (exp11) produced the project's most instructive failure. Greedy teacher scheduling (route each rollout to the most-likely specialist) collapsed completely: within 60 steps, 100% of rollouts routed to one teacher, the other domain decayed to garbage. Reverse KL is mode-seeking; greedy scheduling feeds it a ratchet.
Balanced round-robin scheduling stopped the collapse, but even then, the distilled student did not beat a matched-budget SFT continuation. At specialist-advantage gaps of ~0.24 nats, consolidation's overhead exceeds its value. The honest conclusion is negative and useful: OPD's economics only clear when specialists are strong enough for their logit-level knowledge to outweigh the scheduling noise, which is exactly why the paper pairs OPD with full-vocabulary distillation and dedicated teacher-scheduling infrastructure rather than treating it as a free merge.
9 · The rate nobody prices: determinism¶
Two accidents taught me more than any planned experiment.
First: my MoE implementation updated router-balancing biases during evaluation. Harmless-looking, it broke bitwise reproducibility between identical forward passes, a miniature of exactly why the paper ships batch-invariant, deterministic kernels and treats bitwise alignment across training/inference as a feature, not pedantry.
Second: my agent trajectory runner (exp12) proved that a process killed with SIGKILL mid-trajectory can resume from a write-ahead log and checkpointed sandbox state and finish byte-identical to an uninterrupted run, with non-idempotent commands fast-forwarded, never re-executed. The paper notes that regenerating interrupted rollouts from scratch isn't merely slow; it's mathematically wrong: shorter trajectories survive interruption more often, biasing the training distribution.
When trajectories become training data, crash-consistency determines the dataset. Determinism is the substrate rate all other rates are denominated in.
10 · Reading the final ledger¶
Every number above carries a tag from the project's evidence ledger: PAPER for what arXiv:2606.19348v1 states, MEASURED for what my miniatures actually executed, DERIVED for arithmetic between them, UNKNOWN for what neither establishes. Directions agree throughout; magnitudes never could: synthetic max-entropy sequences are the worst case for compression, and a CPU laptop is not a GPU fleet.
What the exercise changed in me is the reading habit itself. I no longer see components. I see quotes in a currency I can now price:
- CSA/HCA quote memory in units of recall.
- mHC and Muon quote expressiveness in units of stability guarantees.
- Expert-parallel overlap quotes time in units of bandwidth.
- FP4 QAT quotes serving cost in units of training adaptation.
- OPD quotes unity among specialists in units of scheduling discipline.
- The agent sandbox quotes datasets in units of crash-consistency.
DeepSeek-V4 is not a model with efficient parts. It is a set of negotiated exchange rates that happen to compile into a language model, and every rate was forced into existence by the same creditor: a million tokens, due in full, every forward pass.
Built with deepseek-v4-lab: 13 experiments, 65 passing tests, one integrated 629K-parameter chain. Sources: paper · evidence ledger · final artifact map. All diagrams and numbers marked MEASURED come from the repository's own runs; paper-reported figures are tagged PAPER and cited inline.