Course 0: The V4 Thesis

Reading time: 15 minutes · Paper coverage: Abstract, Section 1 Evidence tags used on this page: [PAPER] reported by arXiv:2606.19348v1 · [MEASURED] executed in this repo · [DERIVED] arithmetic from paper facts · [INFERRED] authored interpretation

Four numbers that are easy to misread

DeepSeek-V4 ships as two models:

	total params	activated params	context	tokens seen
V4-Pro	1.6T	49B	1M	33T
V4-Flash	284B	13B	1M	32T

[PAPER Section 4.2.1]

These four quantities are independent axes, and confusing them is the most common failure mode when reading frontier reports:

1. Total parameters measure capacity you own.
2. Activated parameters measure compute you pay per token. A 3% activation ratio is not "a smaller model"; it is a claim that routing can be learned reliably at trillion scale.
3. Context capacity measures what fits, not what is remembered. Capacity, curriculum exposure, and demonstrated retrieval are three different claims.
4. Test-time effort (the report's reasoning-effort modes) measures how long the model thinks, orthogonal to all of the above.

The efficiency headline that actually matters [PAPER Section 1]: at 1M-token context, V4-Pro costs 27% of DeepSeek-V3.2's single-token FLOPs and 10% of its KV cache; it has more activated parameters than any prior DeepSeek model. The architecture did not get cheaper despite growing; it got cheaper because it grew the right dimension (capacity) and shrank the wrong one (per-token memory work).

The central question of this course

What had to change, mathematically, computationally, and operationally, for million-token context to be affordable rather than merely possible?

Everything in this learning system is organized as an answer chain:

million-token demand
  → quadratic attention unacceptable            (Section 2.3 CSA/HCA)
  → compressed KV changes cache geometry        (Section 3.5 heterogeneous cache)
  → sequence partitioning must respect pooling  (Section 3.4.3 contextual parallelism)
  → serving must reuse prefixes on disk         (Section 3.5.2)
  → MoE capacity must route stably              (Section 2.1)
  → communication must hide under compute       (Section 3.1 wave overlap)
  → optimization must not explode logits        (Section 2.4 Muon + RMSNorm'd q/KV)
  → precision must be trained, not bolted on    (Section 5.2.1 FP4 QAT)
  → specialists must consolidate                (Section 5.1 OPD)
  → agent trajectories become training data     (Section 5.2.3–5.2.5)

How to verify claims in this system

Every statement carries exactly one tag:

tag	meaning
PAPER	stated in arXiv:2606.19348v1 (section cited)
DERIVED	arithmetic consequence of paper facts
IMPLEMENTED	exists in src/deepseek_v4_lab with tests
MEASURED	observed by running experiments/exp*
INFERRED	our interpretation
UNKNOWN	neither paper nor lab establishes it

Rule: "the paper reports" never becomes "our experiment proves." Our miniatures run at ~1/2,500,000th of Pro's parameter count; directions transfer, magnitudes do not.

Prerequisites

Skim Concept Prerequisites if any of these are admission tickets rather than vocabulary: causal attention & KV caching, GQA/MQA, mixture-of-experts routing, residual streams, Newton–Schulz iteration, quantization grids, reverse KL divergence.