Post-Training and Agentic Learning
Capability becomes controlled policy
Pretraining creates broad capability. Kimi K3 Section 4 describes the feedback system that turns it into a policy able to reason at controlled effort, operate tools, learn from verified outcomes, consolidate specialist behaviors, and survive deployment arithmetic.
View the retained Course 3 notebook on GitHub
Prerequisite bridge: six ideas before the pipeline
| Concept | Plain-language meaning | Why K3 needs it |
|---|---|---|
| Cold-start policy | A supervised policy that can already follow instructions and use tools before RL begins. | Reward search should refine competent trajectories rather than rediscover language and tool syntax. |
| Partial rollout | Begin an update after enough trajectories finish; preserve and resume the stragglers. | Long agent tasks have heavy-tailed runtimes and would otherwise stall synchronous training. |
| Agentic GRM | A judge agent writes a rubric, scores candidates, and records the evidence. | Many useful general tasks lack deterministic verifiers. |
| MOPD | The student samples its own prefixes; one domain-and-effort teacher supplies dense token feedback. | Nine specialists must become one controllable policy without training only on teacher prefixes. |
| MXFP4/MXFP8 QAT | Expose expert weights and activations to their deployment precision during training. | Rollout and update arithmetic should agree with serving arithmetic. |
| EAGLE-3 | A small recurrent draft proposes tokens that the target model verifies losslessly. | Serving can reduce latency without changing the target distribution. |
Source: Author-created prerequisite synthesis for Kimi K3 §4.1, arXiv:2607.24653v2; external mechanisms are sourced individually in the concept library.
These are different mechanisms. Partial rollout is a scheduler, GRM is a reward source, MOPD is a distillation objective, QAT is numerical training policy, and EAGLE-3 is an inference acceleration path. Their interaction—not a shared acronym—is the Section 4 story. Definitions, dependencies, and original sources are collected in the concept library.
4.1 Method: demonstrate, specialize, consolidate
Three stages solve three learning problems
The paper reports three stages:
- supervised fine-tuning establishes a productive cold-start agent;
- reinforcement learning develops domain specialists at low, high, and max reasoning effort;
- Multi-Teacher On-Policy Distillation consolidates nine specialists into one model.
This is not a sequence of interchangeable optimizers. SFT supplies demonstrations, RL supplies consequences, and MOPD supplies dense specialist guidance on the student’s own prefixes.
4.1.1 Supervised fine-tuning
Verified agent trajectories
Prior Kimi specialists synthesize complex agent trajectories. Those trajectories pass multi-stage verification and human-in-the-loop annotation, then are serialized with the XTML chat template. A trajectory can preserve reasoning, tool calls, observations, recovery, and the final product; its learning value is the action sequence, not merely the answer.
Quantization-aware training begins at SFT
Quantization-aware training begins here, before RL, using MXFP4 expert weights and MXFP8 activations. This makes deployment arithmetic part of post-training rather than a final compression surprise.
4.1.2 Reinforcement learning
Three domains by three effort levels
K3 crosses three domains—general tasks, general agents, and coding agents—with effort levels \(\{\text{low},\text{high},\text{max}\}\), producing nine experts. The report does not define these levels as fixed token counts and does not describe an automatic task-complexity router.
Partial rollout
For \(N\) prompts and \(K\) samples per prompt, the system keeps \(NK\) trajectories active. Generation yields to policy optimization when a fraction \(\lambda\) has completed. Unfinished trajectories are paused and prioritized for resumption in the next iteration. This removes the synchronous straggler barrier, but creates stale off-policy data; per-token regularization constrains updates locally enough to tolerate it.
Reasoning-effort curriculum
Each problem \(x\) receives a cold-start token budget \(b_0(x)\). If trajectory \(y\) uses more than \(\tau b_0(x)\), its reward is overridden:
Source: Kimi K3 §4.1.2, problem-relative reasoning-effort penalty, arXiv:2607.24653v2.
For general tasks, \(T\) counts thinking tokens. For agentic tasks it includes reasoning and tool-call argument tokens. Training begins with a relatively large, capped multiplier for max effort, then anneals \(\tau\) by domain to obtain high- and low-effort policies.
Agentic generative reward model
Non-verifiable general tasks use tournament-style binary comparisons. The judge must read the product, generate a rubric, score each candidate against it, and record the rubric scores in a scorepad. A separate verbosity budget makes overlong candidates automatically lose, limiting a common reward-hacking direction.
4.1.3 Multi-Teacher On-Policy Distillation
Select the teacher by domain and effort
For sampled domain \(d\) and effort \(e\), the matching teacher \(\pi^{(d,e)}_{teacher}\) guides the unified student.
Equation 15: clipped token-level log ratio
On student token \(y_t\) after student prefix \(y_{<t}\), Equation 15 is
Source: Kimi K3 §4.1.3, Eq. (15), arXiv:2607.24653v2.
The stop-gradient makes the reward a fixed signal for the policy update; clipping prevents extreme ratios from dominating. This is dense per-token guidance and integrates with partial rollouts. It is not weighted KL averaging over all nine teachers. The paper reports that a finer top-k objective brought no clear advantage.
4.1.4 Deployment-aware post-training
Quantization-aware post-training
Routed MoE expert weights—most of parameter memory—use MXFP4; their input activations use MXFP8. Attention projections, latent-MoE projections, shared experts, and routers remain at higher precision. RL rollout and training use the same scheme, removing that source of train–inference mismatch.
EAGLE-3 draft fine-tuning
The pretrained MTP layer structurally matches the one-layer draft. The target model is frozen; only the draft and bias-free fusion projection update. Low-, mid-, and high-level features come from the first, fourth, and final AttnRes blocks. Initialization \([0\;0\;I]\) begins from the high-level feature already familiar to MTP. A seven-step unroll trains later draft steps on earlier draft outputs, matching recurrent inference.
Equation 16: optimize acceptance mass
For target distribution \(p\) and draft \(q\), lossless speculative acceptance is
Source: Kimi K3 §4.1.4, Eq. (16), using LK Losses [Samarin et al., 2026].
One token’s journey through MOPD
Suppose the student is sampling a coding-agent, high-effort trajectory. That label selects the matching teacher; the other eight teachers are not averaged in. At token (t), both policies score the student’s actual prefix. If the selected teacher assigns relatively more probability to the sampled token, the clipped log ratio is positive; if it assigns less, the signal is negative. Stop-gradient prevents the reward itself from becoming a differentiable path into the teacher. The policy optimizer then combines this dense token signal with the trajectory-level outcome. This worked trace explains the data flow; it does not reproduce K3’s undisclosed optimizer coefficients or distributed collection system.
LK directly optimizes shared probability mass. Lower KL need not maximize acceptance for a capacity-limited draft.
4.2 Task synthesis and agentic environments
RL requires both diversity and trustworthy evaluation. A persuasive completion claim is not a verified world state.
4.2.1 Unified white-box environment
An agent harness is decomposed into configurable tools, system prompts, context strategies, skills, memories, subagents, and protocols. Different task groups receive different compositions, including known and novel harness configurations. The objective is cross-harness generalization rather than memorization of one schema.
4.2.2 Knowledge-graph-guided synthesis
Agents recursively expand a hierarchical directed acyclic graph from coarse seed domains toward atomic concepts. Before adding a node they inspect the graph for equivalent or related concepts. Related nodes and ancestors form keyword context; web retrieval gathers public source materials; a synthesis agent produces a selected task type.
4.2.3 Verifiable agent problems
Representative families include multi-step evidence search, professional deliverables completed through sandbox tools, and visual reasoning in which Python crops, transforms, measures, or verifies images over multiple turns. The common requirement is an observable product or trace that a reward system can inspect.
4.2.4 Kernel optimization
Tasks range from individual operators to fused mega-kernels across CUDA, Triton, CuTe DSL, Gluon, ThunderKittens, and TileLang. Numerical error beyond threshold gives zero reward. A correct solution matching the expert baseline receives 0.5; movement toward the hardware roofline approaches 1. Replay, input caching, and illicit precision reduction are explicitly detected.
4.2.5 Personal assistants
Mock Gmail, Notion, Slack, and Canvas applications enable reproducible persistent worlds without external APIs. Events unfold across simulated days and applications, each with an evaluation criterion. The paper reports that a single rollout may reach thousands of tool calls and millions of context tokens; these are possible scales, not typical values.
4.2.6 Autonomous Execution Tasks
The agent receives an initial state, goal, constraints, tools, budgets, and verification interfaces—no reference trajectory. It must decompose, act, recover, and decide when to stop. Reward comes from independent evaluation of final state. Public verifiers provide diagnostics; hidden verifiers check held-out scenarios; limited submissions and penalties resist gaming.
4.2.7 Web development
Tasks span websites, games, WebGL scenes, visualization, SVG, and full-stack applications under diverse scaffolds. Deterministic checks test behavior and, where applicable, structural or pixel similarity. Model judges inspect source code and interact with rendered outputs. Build failure, runtime error, or a fake artifact zeros reward.
Synthesis: one policy, many feedback systems
The final policy is jointly shaped by verified demonstrations, specialized outcomes, problem-relative effort budgets, routed teachers, quantization, speculative drafting, task synthesis, and independent verifiers. Post-training is therefore an engineered feedback system, not a final layer of stylistic polish.
Evidence boundary
The repository executes partial-rollout scheduling, effort-budget penalties, Equation 15 token rewards, Equation 16 overlap loss, and DAG validation on small deterministic fixtures. It does not reproduce nine expert models, RL scaling curves, MXFP kernels, speculative speedup, reward-model quality, or the paper’s agent environments.
Retrieval check
Explain why on-policy distillation uses student-generated prefixes, why effort is problem-relative, and why an acceptance calculation in the notebook does not establish production speculative-decoding speedup.