The Whole Machine

The culmination: models/tiny_v4.py · 629K parameters · every mechanism in one trainable stack (Exp13) Toy-scale result. Direction is informative; production magnitude is not.

Every course on this site ends by pricing one mechanism. This page is where they stop being separate. One model, trained end-to-end, containing all seven at once.

What is actually connected

        Muon (matrix params) + AdamW (embeddings, heads, norms)
                          |
        mHC lanes: Birkhoff-constrained residual mixing
                          |
                  attention (hybrid)
                  /                    \
        CSA layers               HCA layers
   m=4 dual-overlap pooling   heavier pooling, dense readout
   lightning indexer, top-k   no selection step
                  \                    /
              sliding-window branch (raw recent tokens)
                          |
              MoE feed-forward
       Sqrt(Softplus) router, bias-only balancing
                          |
                MTP head (depth 1)

Data flows through mHC-weighted lanes into alternating compressed-attention layers, routes through fine-grained experts, and trains against both next-token and multi-token objectives under the Muon/AdamW split.

Integrated vs Simulated vs Not reproduced

The evidence discipline that governs every page governs this one too. Three honest buckets:

Integrated [IMPLEMENTED + MEASURED]

Actually running together in one forward/backward pass:

mechanism	status in tiny_v4
mHC lane mixing	active, Sinkhorn projection in the graph
CSA + lightning indexer	true top-k sparse attention through training, not a relaxed approximation
HCA alternation	interleaved with CSA per the hybrid design
MoE	√Softplus affinities, aux-loss-free bias balancing
MTP head	depth 1, joint loss
Muon + AdamW split	matrix params via Muon, rest via AdamW

Headline result: end-to-end training through true top-k sparse attention succeeds; sparse-phase retrieval loss reaches 0.003 to 0.11 [MEASURED]. Honest cost: sparse eval accuracy is 28.9% (chance 6.25%; the dedicated 2-layer CSA of Exp02 reaches ~88%), because stack complexity makes the indexer's job harder at this budget. We state that limitation rather than tune it away.

Simulated [MEASURED, system model]

Represented by an executable model, not full production machinery:

concern	how it appears here
EP communication overlap	wave-pipeline simulator reproducing the C/B crossover law
KV cache reuse economics	manager implementing the real layout (lcm blocks, state buffer, ring window) with persist/reload timings at CPU scale
FP4 arithmetic	fake quantization on the E2M1 grid with STE; no native FP4 hardware

Not reproduced [UNKNOWN at this scale]

Requires frontier-scale infrastructure; studied and documented, not executed:

concern	why out of reach
trillion-parameter stability tricks (anticipatory routing, clamping)	paper itself marks them empirically-understood only
RL rollout pipeline with reward models and >10 specialists	our OPD study documented a toy-scale negative result instead
Rust sandbox fleet, 3FS storage, EROFS layered images	infrastructure substrate, not mechanisms
GPU kernel determinism/batch-invariance at scale	surfaced here as a bug lesson (eval-time state mutation), not as kernels
native FP4 compute and lossless FP4-to-FP8 dequant	emulated implicitly via per-tile scales

Why the assembly matters

Separately, each page proves one mechanism works. Together they prove something harder: the mechanisms are composable. Compression survives inside mHC lanes; the indexer trains through MoE routing noise; MTP adds its loss without destabilizing Muon updates. That composability is the actual thesis of V4, and the place it could have broken was integration, not isolation. Exp13 is where we looked.

Full artifact map: Final Evaluation · run it yourself: python experiments/exp13_tiny_v4.py