Pretraining the Foundation
Architecture is possibility; training is acquired behavior
Architecture defines possible computation. Pretraining determines the distribution of experience, the allocation of finite compute, the stability of optimization, and the range over which the model learns to operate. Kimi K3 Section 3 is therefore best read as one causal program—not four disconnected implementation notes.
View the retained Course 2 notebook on GitHub
3.1 Pre-Training Data: tokens are designed
What each source family contributes
- Web Text supplies breadth and linguistic variation.
- Code supplies executable structure and dense procedural signal.
- Mathematics supplies symbolic transformations and proof patterns.
- Knowledge supplies reference-style factual and explanatory material.
- Vision supplies perceptual, spatial, temporal, OCR, and code-to-render relationships.
These are pedagogical descriptions of signal, not claims that a domain deterministically produces a capability. The paper reports the taxonomy and smaller-model mixture ablations; it does not publish final mixture weights.
Kimi K3 reports four principal text domains—Web Text, Code, Mathematics, and Knowledge—plus a large vision corpus covering captions, interleaved image–text documents, OCR, perception, video, and visual coding.
Filtering edits the learning distribution
Each domain uses rule-based filters and classifier quality scores. A rule can reject malformed structure before expensive scoring; a learned classifier can estimate a domain-specific quality signal. Retaining the decision and its provenance matters because every acceptance or rejection changes the distribution seen by the optimizer.
Deduplication controls repeated pressure
Exact hashes catch normalized copies while fuzzy similarity catches altered copies. Without both, repetition silently becomes an unintended sampling weight. Aggressive thresholds introduce the opposite risk: erasing legitimate recurring templates.
Rephrasing must preserve meaning
Knowledge and mathematics material is also rephrased using diverse styles and perspectives, chunk-wise autoregressive generation, and source-fidelity verification. Vision data combines open collections with internal filtering, synthesis, and deduplication; coordinate supervision uses both absolute and normalized [0,1] formats. Programmatic multimodal pairs connect source code to SVG, 3D, webpages, games, and CAD renderings.
Rephrasing has a three-part contract: preserve a source as the meaning anchor, vary the surface presentation, and reject transformations that fail fidelity verification. Diversity without the third step could manufacture fluent corruption.
Vision is a family of supervision problems
Captions teach description, OCR connects pixels with written language, perception teaches localization, video adds temporal change, interleaved documents demand cross-modal reference, and visual coding connects editable programs to visible results. Calling all of this “vision data” hides the distinct learning problems.
Programmatic vision links cause and consequence
Programmatic pairs add a concrete causal relationship: code states construction intent and the renderer exposes its visual consequence. Normalized coordinates \(u=x/W,\;v=y/H\) make localization comparable across resolutions and complement absolute coordinates.
Mixture weights are curriculum policy
This matters because the post-curation distribution is the optimizer's world. Quality filters can remove noise and also erase rare expertise. Deduplication reduces repeated gradient pressure and can also collapse legitimate templates. Mixture weights are therefore a policy:
Source: Author-created mixture notation for the data blending described in Kimi K3 §3.1; the paper does not publish the final mixture weights.
The local miniature records accepted, too_short, low_quality, exact_duplicate, and near_duplicate decisions and samples a deterministic weighted mixture. It demonstrates mechanics, not the quality or composition of K3's corpus.
Evidence boundary
The domain taxonomy and pipelines above are paper-reported. The repository fixture contains six tiny authored examples. Passing its assertions does not validate internet-scale filtering, classifier calibration, perceptual video deduplication, or downstream model quality.
3.2 Scaling Law: compute is a constraint
Observe small, then fit the scaling surface
Scaling begins with measured validation losses from controlled smaller runs. The smooth surface is inferred from those noisy observations; it is not a physical law supplied in advance.
The iso-compute tradeoff
A common teaching approximation writes training compute as
Source: Standard dense-Transformer teaching approximation used to frame Kimi K3 §3.2; not a K3 paper equation and not exact for its sparse multimodal architecture.
where \(N\) is activated model parameters and \(D\) is training tokens. A synthetic loss surface can expose the allocation tension:
Source: Standard scaling-law form used pedagogically for Kimi K3 §3.2; the local coefficients are authored and are not a fit to the paper’s Fig. 7.
Along a fixed-compute curve, a larger model leaves fewer tokens; more tokens require a smaller model. The operating point balances both fitted loss terms rather than maximizing one axis.
Four coupled choices must be retuned
K3 reports dedicated studies retuning batch size, learning rate, tokens per parameter, and model shape on held-out OOD validation data. It reports an approximately 2.5× scaling-efficiency gain over K2. This is a paper result, not reproduced here.
Why cosine versus WSD requires two searches
K3 reports that cosine decay consistently beats Warmup Stable Decay under each schedule's independently optimized hyperparameters. Peak learning rate and batch size differ substantially between the schedules, even at fixed model size and token budget. Reusing one shared configuration would test configuration alignment, not the schedules fairly.
The fair comparison is therefore
Source: Author-created fair-comparison notation for the independently tuned schedule study in Kimi K3 §3.2; not a paper equation.
not two schedules evaluated under one inherited configuration.
Read scaling efficiency horizontally
Figure 7's approximately 2.5× claim is a horizontal compute comparison at a target loss, \(C_{K2}(L^*)/C_{K3}(L^*)\approx2.5\). Its reciprocal is roughly 0.4, so “about 40% of the compute at equal fitted loss” is a useful interpretation—not an independently reproduced result.
The notebook's smooth surface uses authored coefficients solely to make iso-compute arithmetic executable. It is not a fit to Fig. 7 and must not be used to infer K3 scale or loss.
3.3 Training Recipe: a synchronized control system
Native multimodal next-token prediction
K3 trains language and vision jointly from the beginning with one interleaved next-token prediction objective. This is native multimodal pretraining, not a language model followed by a post-hoc alignment stage.
Stability is a coordinated control stack
The reported control stack is:
- Per-Head Muon for matrix parameters;
- K2-style weight clipping;
- Quantile Balancing for MoE load;
- cosine learning-rate decay;
- 1% linear warmup;
- 0.1 weight decay;
- initial 8K context, later extended to 64K during pretraining.
Each control acts on a different failure surface: Per-Head Muon changes update geometry, clipping limits magnitude, Quantile Balancing changes expert selection thresholds, and the schedule controls step size.
Per-Head Muon preserves local update geometry
Full-matrix Muon couples the concatenated projection; K3 instead applies Newton–Schulz to each head's momentum block, \(\Delta W_h=\operatorname{NS}(M_h)\). The rationale concerns different gradient or momentum scales—not differently sized heads.
For peak rate \(\eta_{\max}\), minimum rate \(\eta_{\min}\), warmup endpoint \(t_w\), and total steps \(T\), the local schedule miniature implements
Source: Standard cosine schedule implemented by the local miniature to explain Kimi K3 §3.3; exact production schedule parameters beyond those reported are unavailable.
That assertion verifies endpoints and warmup arithmetic. It does not establish loss stability or reproduce the full training run.
3.4 Long-Context Extension: capacity is not capability
Three claims must remain separate:
- Architectural capacity: the network accepts a sequence of a given length.
- Training exposure: optimization includes examples and dependencies at that length.
- Demonstrated capability: evaluation establishes retrieval or reasoning across that range.
No positional rescaling
K3 uses NoPE and encodes order implicitly through KDA recurrence, gating, and decay. The paper therefore reports direct extrapolation to 1M without RoPE interpolation or rescaling. That removes one positional-encoding intervention; it does not by itself prove that distant information is retained.
Recency emerges through repeated recurrent state updates. The model does not explicitly assign one decay value to “old tokens” and another to “recent tokens.” NoPE means no explicit positional embedding—not no sequence order.
Long data must be cleaned and made causally useful
The paper reports exact and fuzzy deduplication for long documents, perceptual frame hashing for video, heuristic and classifier filters, and structural validation. Long coherent sources are upsampled during cooldown because short sequences would otherwise dominate.
Length alone invites local shortcuts. K3 additionally reports permuting and concatenating multimodal documents and subtasks so solutions require evidence scattered across the full context. The educational distinction is crucial: padding creates distance; task construction creates dependence.
A million-token tensor can still contain a locally solvable objective. A stronger example places necessary clues at separated locations so that omitting any one changes the answer. This converts physical distance into causal dependence.
Four-stage curriculum
Source: Kimi K3 §3.4, reported staged context extension, arXiv:2607.24653v2.
The first extension occurs during pretraining; 256K and 1M occur during cooldown. Concentrating expensive long-sequence computation late keeps the program economical. Sequence partitioning for KDA is covered in the infrastructure course (§5.1.2).
The repository uses illustrative stage boundaries to animate the ordered curriculum and asserts that chunking preserves every token. Those boundaries are not reported by the paper, and tensor preservation is not semantic retention.
Synthesis
The Section 3 story is a dependency chain:
flowchart LR
A[Curated domains] --> B[Mixture policy]
B --> C[Scaling observations]
C --> D[Compute allocation]
D --> E[Stable recipe]
E --> F[Progressive context curriculum]
F --> G[Long-context foundation]Source: Author-created synthesis of the training program reported across Kimi K3 §§3.1–3.4; not a paper figure.
Changing an upstream decision changes everything downstream. That is why architecture diagrams alone cannot explain the trained model—and why every local miniature is labeled separately from the full-scale claims reported by Kimi K3.
Retrieval check
Predict what breaks if long-context stages are introduced without long-source cleaning, and explain why a scaling-law fit cannot independently validate the paper's reported 2.5× efficiency result.