Course 4: General Infrastructures¶
Paper coverage: Sections 3.1–3.3 Lab:
dist/comm_sim.py· exp08
Infrastructure is where V4 stops being mathematics and starts being a machine. Three systems in this chapter answer one question from three angles: what does the model owe the hardware, and what can the hardware forgive?
Evidence tags: PAPER stated by DeepSeek · DERIVED mathematically implied · IMPLEMENTED reproduced in this project · MEASURED observed in our experiments · INFERRED our interpretation · UNKNOWN not established. Full definitions: evidence ledger
1 · Expert-parallel overlap: paying the post office with idle time¶
The mental model: every MoE layer runs an internal postal service. Tokens are packed (Dispatch), shipped to whichever rank owns their expert, processed, and shipped back (Combine). Naively, the model stops computing while mail moves, you pay twice for every layer.
V4's Section 3.1 starts from a measurement, not an idea: inside one MoE layer, total communication time is less than total computation time. That single fact changes everything downstream. If mail time fits inside work time, then a scheduler that overlaps them continuously never starves the GPU; compute becomes the only bottleneck that matters.
Making this continuous requires waves: experts are grouped so that while wave i computes, wave i+1 is being dispatched and wave i−1 is being combined, a three-deep pipeline that never drains. Dispatch is pull-based (receivers read remote activations) because fine-grained push notifications cost more latency than the data is worth.
The design rule worth memorizing:
For V4-Pro's SwiGLU experts: 6·h·d FLOPs per token-expert pair against ≈3·h bytes shipped, so \(C/B \le 2d = 6144\) FLOPs per Byte of interconnect. Below that ratio the network waits; above it, bandwidth has no further value; silicon should go elsewhere.
In the lab: our simulator reproduces exactly this shape: speedup saturates at the crossover; exposed communication grows linearly below it [MEASURED exp08]. The law transfers; the constant is just arithmetic.
2 · TileLang: kernels as a product, not a project¶
The problem behind Section 3.2: a model this fused needs hundreds of small kernels. Hand-written CUDA makes each one a week; unfused operators drown in launch overhead. TileLang is DeepSeek's answer, a Python-embedded DSL where one codebase prototypes and ships production kernels.
Three ideas carry the section:
- Host codegen: runtime checks and argument marshalling move out of Python into generated host code. Per-invocation CPU overhead drops from tens/hundreds of microseconds to under one microsecond, which is what makes many small kernels viable at all.
- Z3 in the compiler, tile layouts, memory hazards, vectorization choices become formal integer-arithmetic queries (QF-NIA), verified rather than guessed.
- Numerics as policy: fast-math disabled by default, IEEE-exact intrinsics opt-in. Bitwise reproducibility versus raw speed is a choice the programmer makes explicitly, not a compiler accident.
The transferable lesson: kernel productivity tooling is what lets an architecture team iterate on attention variants weekly instead of quarterly [INFERRED from the paper's emphasis].
3 · Batch-invariant & deterministic kernels: same input, same bits, everywhere¶
Why would anyone pay performance for bit-exactness? Because trillion-parameter training produces loss spikes, and you cannot debug a spike if re-running the same step gives different numbers. And because RL/post-training pipelines compare probabilities across engines, bitwise agreement between training and serving eliminates a whole class of silent drift.
What breaks it, and the fixes [PAPER Section 3.3]:
| non-determinism source | fix |
|---|---|
| split-KV attention (output depends on SM count) | forbid it; dual-kernel decoding with identical accumulation order |
| cuBLAS heuristics | replace with DeepGEMM end-to-end; drop split-k mostly |
atomicAdd in backward |
per-SM accumulation buffers + deterministic global reduce |
| concurrent MoE backward writes | token-order preprocessing + per-rank buffer isolation |
Lab echo: we hit this wall ourselves. Our toy MoE updated router-balancing biases during evaluation, a state mutation nobody noticed until two identical forward passes produced different logits (caught by a test after integration). The paper spends an entire section on this class of bug; now we know why firsthand [MEASURED].