architecture · design-space · three-axes |
Animated Interpretation |
A single hidden state must carry history, retrieve useful depth, and select specialized capacity. |
§2 |
This organizing frame is a teaching synthesis, not a decomposition named by the paper. |
architecture · native-multimodality · patch-project-merge |
Animated Interpretation |
Pixels become patches, visual features, projected embeddings, and finally members of the same sequence as text. |
§2.4 |
Local evidence validates patch and shape flow, not MoonViT-V2 training or multimodal quality. |
architecture · hybrid-cadence · execute-and-replicate-motif |
Paper Reported |
One token traverses three recurrent updates and one global read; twenty-three motifs plus a final MLA produce 69 KDA and 24 MLA layers. |
§2.1 |
The animation explains layer cadence; it does not measure the quality or cost contribution of the ratio. |
architecture · kda · recurrence |
Executed Here |
KDA replaces a growing list with a fixed state, then decays, probes, corrects, writes, reads, and resolves into the notebook trace. |
§2.1.1, Eqs. 1–6 |
The sequential miniature verifies the recurrence and chunk handoff, not FlashKDA or million-token retention. |
architecture · gated-mla · compress-read-and-gate |
Animated Interpretation |
Compact latents persist; a global query reconstructs head-specific keys and values on demand and gates the result back into the residual stream. |
§2.1.2 |
The miniature checks dimensions and cadence but does not reproduce K3's trained cache-quality tradeoff. |
architecture · attnres · retrieve-over-depth |
Executed Here |
Ordinary addition collapses depth history; AttnRes scores addressable block summaries and forms a depth-selective blend. |
§2.2, Eqs. 8–10 |
Local tests verify normalized depth weights, not full-model optimization or Block AttnRes memory savings. |
architecture · stable-latent-moe · route-stabilize-and-balance |
Animated Interpretation |
The paper combines 896/top-16 latent routing, a shared full-width path, bounded SiTU-GLU, and score-margin Quantile Balancing. |
§2.3–§2.3.3 |
The local 8-expert/top-2 path omits latent/shared branches and uses count-EMA rather than paper Quantile Balancing. |
architecture · per-head-muon · split-orthogonalize-and-repack |
Executed Here |
Split the fused gradient into head-local matrices, apply five polynomial Newton-Schulz steps independently, and repack the original shape. |
§2.5 |
Small-matrix singular-value traces validate mechanics only; rectangular targets are shape-aware semi-orthogonal factors. |
architecture · synthesis · rapid-replay |
Animated Interpretation |
One token passes through multimodal entry, hybrid sequence mixing, depth routing, sparse capacity, and the output head. |
§2 |
The synthesis composes mechanisms conceptually and is not an executed end-to-end K3 model. |
evaluation · measurement-contract · evaluation-thesis |
Animated Interpretation |
A score is the final link in a measurement chain—not a property floating above its protocol. |
§6 |
The chain is an authored evaluation-literacy framework. |
evaluation · measurement-contract · four-axes |
Paper Reported |
Reasoning, coding, agency, and vision are different lenses; none is a universal intelligence scalar. |
§6.1.1 |
Coverage breadth does not prove complete construct coverage or independence among benchmarks. |
evaluation · measurement-contract · baseline-contract |
Paper Reported |
Maximum effort is aligned where possible, but fallback, cyberguard, and xhigh qualifications remain part of the comparison. |
§6.1.2 |
The paper notes fallback and cyberguard behavior; identical model-side conditions are not guaranteed. |
evaluation · measurement-contract · protocol |
Paper Reported |
Model × effort × harness × tools × sampling × task version produces the observed result. |
§6.1.3 |
The report provides substantial configuration detail but not a complete local reproduction package. |
evaluation · measurement-contract · metric-literacy |
Animated Interpretation |
Percentage, F1, Elo, criterion pass rate, and pass@5 answer different questions. |
Table 2 |
The metric taxonomy explains the table; it is not an additional paper result. |
evaluation · public-results · reasoning-profile |
Paper Reported |
Competitive graduate reasoning does not erase the measured research-reasoning gap. |
§6.1.4; Table 2 |
The paper identifies the gap but does not establish a causal mechanism for it. |
evaluation · public-results · coding-profile |
Paper Reported |
Coding performance is strong but task-dependent: K3 leads ProgramBench and SWE-Marathon, while Fable 5 leads FrontierSWE. |
§6.1.4; Table 2 |
Harness selection, H20 task calibration, fallbacks, and benchmark versions are material context. |
evaluation · public-results · agentic-profile |
Paper Reported |
K3 leads several verifiable task suites yet trails the strongest model on Elo-rated knowledge work and harder computer use. |
§6.1.4; Table 2 |
The verified-versus-judged grouping is explanatory; benchmark construction differs in more than one dimension. |
evaluation · public-results · vision-tools |
Paper Reported |
Python changes the evaluated system—and the paired lift reveals where tool-mediated inspection helps. |
§6.1.4; Table 2 |
Paired score lift shows association under the tool protocol, not which internal reasoning step caused it. |
evaluation · public-results · fingerprint |
Paper Reported |
The honest summary is a vector of relative strengths—not 'third overall' or a count of wins. |
§6.1.4 |
The report does not publish a single cross-benchmark aggregate from which a universal overall rank can be computed. |
evaluation · internal-and-cyber · internal-loop |
Paper Reported |
Public benchmarks compare; refreshed internal suites diagnose and steer iteration. |
§6.2.1 |
Private tasks, rubrics, and raw outputs are not independently auditable from this repository. |
evaluation · internal-and-cyber · internal-profile |
Paper Reported |
Internal results sharpen both sides of the profile: orchestration and research strengths; behavior, MIRA, continuous assistants, and agentic vision gaps. |
§6.2.1; Tables 3–4 |
These developer-run benchmarks should be read with their harness assignments, refusals, and private-method caveats. |
evaluation · internal-and-cyber · cyber-tiers |
Paper Reported |
Discovery and end-to-end exploitation are distinct capability tiers with different risk and completion demands. |
§6.2.2 |
Selective human review, excluded refusing baselines, private targets, and the paper's lower-bound warning constrain interpretation. |
evaluation · internal-and-cyber · cyber-failures |
Paper Reported |
The remaining gap is not one number: final-chain completion, strategy under mitigations, debugging loops, and weak verification recur. |
§6.2.2 |
These failure attributions come from the paper's internal trajectory analysis, not this repository. |
evaluation · triangulation-and-value · third-party |
Paper Reported |
Intelligence index, industry tasks, and human-preference arenas are three different external lenses. |
§6.3; Table 5 |
The report cites these third-party snapshots; rankings and Elo-style values drift over time. |
evaluation · triangulation-and-value · pareto |
Paper Reported |
A model is cost-efficient when no alternative is both cheaper and better for the same evaluated workload. |
§6.4; Figure 13 |
Costs combine internal measurements and cited API prices under task-specific harnesses as of July 2026. |
evaluation · triangulation-and-value · why-gaps |
Animated Interpretation |
Observed gap → paper-stated factor → plausible mechanism → speculation: do not collapse these levels. |
§4; §6 |
Cross-section causal explanations are hypotheses unless the paper reports an ablation or direct trajectory analysis. |
evaluation · triangulation-and-value · evaluation-synthesis |
Animated Interpretation |
Measure precisely, compare locally, triangulate independently, explain cautiously, and stop where the evidence stops. |
§6 synthesis |
This final discipline is an authored synthesis of the section's evidence structure. |
evaluation · case-studies · case-method |
Animated Interpretation |
A case becomes evidence only while its task, trajectory, artifact, verification, and boundary remain attached. |
§7 synthesis |
The trajectory schema is an authored case-study reading method. |
evaluation · case-studies · kernel-case |
Paper Reported |
Kernel optimization closes on correctness and latency under the same workload and sandbox. |
§7 |
The measurements and implementation are not reproduced in this repository. |
evaluation · case-studies · compiler-case |
Paper Reported |
MiniTriton matters because frontend, lowering, runtime, autograd, distributed execution, and workload validation compose. |
§7 |
Source, rooflines, and GPT parity are paper-reported and not audited here. |
evaluation · case-studies · chip-case |
Paper Reported |
The nano-KPU case propagates a design through RTL and an open EDA flow to timing, area, and throughput outputs. |
§7 |
The 48-hour case and EDA outputs are not independently reproduced. |
evaluation · case-studies · science-case |
Paper Reported |
Scientific coding remains trustworthy only when sources, assumptions, equations, numerical checks, and interpretation stay linked. |
§7 |
Speed and code volume do not establish scientific correctness; no independent domain audit was performed here. |
evaluation · case-studies · knowledge-case |
Paper Reported |
Knowledge work ends at a reviewable artifact, not at fluent synthesis. |
§7 |
Selected artifacts do not estimate reliability or source completeness. |
evaluation · case-studies · creative-case |
Paper Reported |
The architecture explainer and the teaser assembled from 56 source clips are pipelines of planning, code, rendering, inspection, and revision—not one-prompt films. |
§7 |
The reported artifacts and production trajectories are not reproduced in this repository. |
evaluation · conclusion · supported-conclusion |
Animated Interpretation |
Preserve what is reported, what this repository executes, and what remains an open question. |
§8 |
The synthesis is evidence-bounded; full-scale training and evaluation are not independently reproduced here. |
infrastructure · thesis · four-regimes |
Animated Interpretation |
Training, prefill, rollout, and serving stress different boundaries of the same architecture. |
§5 |
The four-regime framing is an authored synthesis. |
infrastructure · distributed-primer · parallelism-axes |
Animated Interpretation |
DP splits samples, PP splits depth, EP splits experts, and CP splits one sequence. |
Background required by §5.2 |
This coordinate-system diagram is an authored prerequisite synthesis, not a published K3 topology. |
infrastructure · distributed-primer · zero-pipeline-schedule |
Animated Interpretation |
Virtual stages reduce bubbles; 1F1B changes activation lifetimes; ZeRO partitions model-state ownership. |
Prerequisite synthesis for §5.2 and paper-reported configuration |
The visual teaches semantics and memory ownership; it does not reproduce K3's full device mesh or communication schedule. |
infrastructure · kda-systems · flash-kda-regimes |
Paper Reported |
One recurrence requires different schedules in training, long prefill, and token-by-token decoding. |
§5.1.1 |
No local GPU kernel or performance benchmark reproduces FlashKDA. |
infrastructure · kda-systems · kcp-prefix-scan |
Paper Reported |
Each segment becomes an affine map; associative composition reveals every rank's incoming state. |
§5.1.2 |
The notebook verifies local affine composition; it does not implement distributed prefix-scan communication or KCP. |
infrastructure · three-trillion-pretraining · three-bottlenecks |
Animated Interpretation |
Expert skew, activation residency, and variable vision compute are independent bottlenecks. |
§5.2 |
The causal grouping is a teaching synthesis of the paper's three subsections. |
infrastructure · three-trillion-pretraining · moonep |
Paper Reported |
Bounded redundant experts turn variable routing into equal token counts and static launch shapes. |
§5.2.1 |
The notebook verifies equal-capacity overflow arithmetic; it does not implement MoonEP replica placement, dispatch, or zero-copy networking. |
infrastructure · three-trillion-pretraining · memory-ledger |
Paper Reported |
Memory is a lifetime-scheduling problem, not one monolithic allocation. |
§5.2.2 |
The notebook computes a local lifetime ledger; it does not perform recomputation, pooling, offload, gradient transformation, or communication. |
infrastructure · three-trillion-pretraining · overlap-and-reuse |
Paper Reported |
Known expert offsets enable direct placement and GEMM overlap; non-overlapping lifetimes enable safe buffer reuse. |
§5.2.1–§5.2.2 |
The timeline is explanatory; no local multi-GPU transport, allocator aliasing, or P2P Muon benchmark is executed. |
infrastructure · three-trillion-pretraining · vision-critical-path |
Paper Reported |
Dynamic context groups shorten large-image work; pipeline decomposition hides it outside the critical path. |
§5.2.3 |
No local distributed vision pipeline reproduces the reported optimization. |
infrastructure · three-trillion-pretraining · decoupled-encoder |
Paper Reported |
DEP lets variable vision work advance independently, then joins the language path at an explicit feature boundary. |
§5.2.3 |
No local encoder service, queue, or distributed critical-path measurement reproduces DEP. |
infrastructure · million-token-rl · rollout-state-pool |
Paper Reported |
Rollout scheduling can pause trajectories only if KV and KDA state survive outside the active worker. |
§5.3.1 |
The repository has no external cache pool or distributed RL runtime. |
infrastructure · million-token-rl · rollout-control-loop |
Paper Reported |
Memory and queue pressure reduce active decode; exact state preservation makes that control action reversible. |
§5.3.1 |
The feedback equation is explanatory; the paper runtime's controller parameters and production traces are not reproduced. |
infrastructure · million-token-rl · agentenv |
Paper Reported |
AgentENV treats an environment as durable world state with isolation and lifecycle operations. |
§5.3.2 |
The local Python dictionary stores caller data; it is not a microVM or copy-on-write system. |
infrastructure · million-token-rl · overlaybd-storage |
Paper Reported |
OverlayBD separates a shared immutable base from small trajectory-specific changes. |
§5.3.2 |
The repository does not integrate OverlayBD, a block device, or microVM snapshots. |
infrastructure · online-serving · hybrid-prefix-cache |
Paper Reported |
Fine MLA hashes and coarse physical pages remain consistent with sparse KDA checkpoints. |
§5.4.1 |
The notebook checks exact boundary agreement; it does not implement the joint physical cache or state reconstruction. |
infrastructure · online-serving · prefix-reuse-walkthrough |
Animated Interpretation |
A request hits a fine hash, loads coarse pages, restores the nearest KDA checkpoint, and replays only the suffix. |
Mechanism synthesis for §5.4.1 |
The packing and replay diagram explains the reported design without claiming the paper's undisclosed byte layout or hit-rate policy. |
infrastructure · online-serving · serving-kernels |
Paper Reported |
Production serving optimizes the whole hybrid path, including state reconstruction after speculative verification. |
§5.4.2 |
Local code contains no production GPU kernels or measured serving speedup. |
infrastructure · online-serving · kernel-toolchain |
Animated Interpretation |
DSA sparsifies token interactions; KDA recurrently compresses history; cuBLAS and MiniTriton solve different kernel layers. |
Prerequisite comparison for §5.4.2 and §7.1 |
No local DSA, MiniTriton compiler, custom CUDA kernel, or comparative GPU benchmark is executed. |
infrastructure · online-serving · fleet-scheduling |
Paper Reported |
The scheduler weighs locality, memory pressure, latency budget, and fairness before admitting work. |
§5.4.3 |
The notebook filters and scores two fixture nodes; it does not reproduce the distributed fleet scheduler or its measured policy. |
infrastructure · synthesis · infrastructure-synthesis |
Animated Interpretation |
The through-line is control over where state lives, when work runs, and how boundaries remain consistent. |
§5 synthesis |
The systems-level causal chain is an authored synthesis. |
posttraining · pipeline · thesis |
Animated Interpretation |
Post-training changes how capability is elicited, verified, budgeted, and deployed. |
§4 |
The causal framing is an authored synthesis. |
posttraining · pipeline · three-stage |
Paper Reported |
Demonstrate, improve through outcomes, then consolidate. |
§4.1 |
Full-scale training is not reproduced. |
posttraining · sft · trajectories |
Paper Reported |
A demonstration is a verified sequence of reasoning, tools, observations, and outcomes. |
§4.1.1 |
Dataset scale and acceptance rates are not disclosed. |
posttraining · sft · qat-start |
Paper Reported |
Deployment arithmetic enters before reinforcement learning begins. |
§4.1.1; §4.1.4 |
Local code does not emulate MX formats. |
posttraining · rl · expert-grid |
Paper Reported |
Three domains times three effort levels yields nine teachers. |
§4.1.2 |
The paper does not publish fixed token counts for effort levels. |
posttraining · rl · partial-rollout |
Executed Here |
Optimization begins when a fraction lambda completes; unfinished trajectories resume later. |
§4.1.2 |
The local queue demonstrates scheduling, not distributed RL. |
posttraining · rl · effort-control |
Executed Here |
Effort is trained by penalizing trajectories that exceed a problem-relative budget. |
§4.1.2 |
Local budgets are illustrative; K3 values are not disclosed. |
posttraining · rl · grm |
Paper Reported |
The judge must externalize its rubric before comparing candidates. |
§4.1.2 |
Judge accuracy is not reproduced. |
posttraining · mopd · teacher-selection |
Paper Reported |
Each sampled domain and effort selects one matching teacher. |
§4.1.3 |
Nine paper models are not available locally. |
posttraining · mopd · token-reward |
Executed Here |
Every student token receives a clipped teacher-versus-student log-ratio reward. |
§4.1.3; Eq. 15 |
The local tensor calculation does not reproduce convergence. |
posttraining · deployment · qat |
Paper Reported |
Expert weights use MXFP4; activations use MXFP8; sensitive non-expert modules stay higher precision. |
§4.1.4 |
Memory savings depend on complete serving layout. |
posttraining · deployment · eagle |
Paper Reported |
A pretrained MTP layer becomes a seven-step recurrent EAGLE-3 draft. |
§4.1.4 |
Draft speedup is not reproduced. |
posttraining · deployment · lk-loss |
Executed Here |
LK directly optimizes the acceptance probability used by lossless speculative sampling. |
§4.1.4; Eq. 16 |
Local distributions are authored fixtures. |
posttraining · environments · white-box |
Paper Reported |
Vary tools, prompts, memory, context, skills, and subagents by task group. |
§4.2.1 |
No local harness-composition runtime reproduces this mechanism. |
posttraining · environments · knowledge-graph |
Executed Here |
Fine concepts guide source retrieval; related nodes create cross-concept tasks. |
§4.2.2 |
No web-scale expansion is reproduced. |
posttraining · environments · verifiable |
Paper Reported |
Reward must depend on observable outcomes rather than persuasive self-report. |
§4.2.3 |
Evaluator reliability remains a separate concern. |
posttraining · task-atlas · kernels |
Paper Reported |
Wrong kernels receive zero; correct kernels are scored against expert speed and the roofline. |
§4.2.4 |
No GPU benchmark executes in this course. |
posttraining · task-atlas · assistants |
Paper Reported |
Persistent mock applications create interdependent events across simulated days. |
§4.2.5 |
The paper reports up to thousands of calls and millions of context tokens, not a typical case. |
posttraining · task-atlas · autonomous |
Paper Reported |
The agent sees the objective and interfaces—not a reference procedure. |
§4.2.6 |
Verifier isolation reduces but cannot eliminate reward hacking. |
posttraining · task-atlas · webdev |
Paper Reported |
A project that fails, errors, or fakes the artifact receives zero reward. |
§4.2.7 |
Internal reward-model quality is not reproduced. |
posttraining · synthesis · synthesis |
Animated Interpretation |
A deployable agent is downstream of its demonstrations, rewards, budgets, teachers, and verifiers. |
§4 synthesis |
The causal chain is an authored synthesis. |
pretraining · thesis · causal-program |
Animated Interpretation |
The final checkpoint is the residue of every upstream decision. |
§3 |
Causal framing is a teaching synthesis. |
pretraining · corpus · source-domains |
Paper Reported |
Different domains teach different regularities; provenance survives curation. |
§3.1 |
Mixture values are not disclosed. |
pretraining · corpus · filtering |
Executed Here |
A filter reshapes the gradient distribution. |
§3.1 |
Local thresholds are illustrative. |
pretraining · corpus · deduplication |
Executed Here |
Frequency is influence; duplicates create repeated pressure. |
§3.1 |
Authored fixtures do not validate internet-scale recall. |
pretraining · corpus · rephrasing |
Paper Reported |
Variation is useful only when meaning survives. |
§3.1 |
No local generator reproduces the pipeline. |
pretraining · corpus · vision-taxonomy |
Paper Reported |
Seeing, reading, locating, and tracking are different learning problems. |
§3.1 |
Taxonomy is reported, not locally trained. |
pretraining · corpus · programmatic-vision |
Paper Reported |
The code states intent; the render exposes consequence. |
§3.1 |
Render generation is not reproduced. |
pretraining · corpus · mixture-policy |
Executed Here |
The sampler decides where the next unit of compute goes. |
§3.1 |
K3 mixture weights remain undisclosed. |
pretraining · scaling · observations |
Animated Interpretation |
The curve is earned from measurements. |
§3.2 |
Local coefficients are synthetic. |
pretraining · scaling · iso-compute |
Executed Here |
Finite compute cannot maximize model size and data simultaneously. |
§3.2 |
C approximately 6ND is a teaching approximation. |
pretraining · scaling · coupled-knobs |
Paper Reported |
Change the model family and yesterday’s recipe stops being evidence. |
§3.2 |
Full-scale optima are not reproduced. |
pretraining · scaling · fair-schedules |
Paper Reported |
Shared hyperparameters test compatibility, not method quality. |
§3.2 |
Schedule searches are not reproduced. |
pretraining · scaling · efficiency |
Paper Reported |
Compare compute at equal loss, not raw parameter counts. |
§3.2; Fig. 7 |
The ratio is reported, not independently reproduced. |
pretraining · recipe · native-objective |
Paper Reported |
One stream, one loss, one shared backbone. |
§3.3 |
Full training is not reproduced. |
pretraining · recipe · control-stack |
Paper Reported |
No single mechanism owns stability. |
§3.3 |
Only schedule arithmetic executes locally. |
pretraining · recipe · per-head-muon |
Paper Reported |
Normalize update scale where the architecture is modular. |
§2.5; §3.3 |
The full derivation belongs to Course 1. |
pretraining · long-context · nope |
Paper Reported |
Removing positional rescaling removes one intervention, not the learning problem. |
§3.4 |
NoPE alone does not prove useful million-token retention. |
pretraining · long-context · distant-evidence |
Paper Reported |
Padding creates distance; task design creates dependence. |
§3.4 |
Local chunk preservation is not semantic retention. |
pretraining · long-context · curriculum |
Paper Reported |
Spend expensive long-sequence tokens late and deliberately. |
§3.4 |
Local progress boundaries are illustrative. |
pretraining · synthesis · causal-synthesis |
Animated Interpretation |
Architecture supplies possibility; pretraining acquires behavior. |
§3 synthesis |
The causal chain is an authored synthesis. |
thesis · two-scaling-axes · plot-two-axes |
Animated Interpretation |
Pretraining changes the policy you begin with; test-time scaling changes how much work that policy can spend on this problem. |
§1 |
The two-axis diagram is an authored organizing model; it is not a reported ablation. |
thesis · model-facts · separate-scale-facts |
Paper Reported |
2.8T total parameters, 104B active parameters, and a 1M-token context window are not one scalar called scale. |
§1 |
These are paper-reported model properties, not independently reproduced measurements. |
thesis · contribution-map · pretraining-axis |
Animated Interpretation |
Architecture, data, compute allocation, optimization, and context curriculum create the starting policy. |
§1; §§2–3 |
This dependency chain is a teaching synthesis, not a component ablation. |
thesis · contribution-map · test-time-axis |
Animated Interpretation |
Useful test-time scale requires learned effort, stateful environments, verification, and infrastructure. |
§1; §§4–5 |
The diagram combines reported mechanisms without assigning measured causal contribution. |
thesis · contribution-map · assemble-paper-map |
Animated Interpretation |
Architecture creates the computation; pretraining shapes the foundation; post-training creates effortful agents; infrastructure makes both feasible; evaluation bounds the claim. |
§1; §§2–8 |
The dependency chain is a teaching synthesis and does not imply a measured causal contribution for every component. |
thesis · evidence-contract · label-evidence-levels |
Animated Interpretation |
Reported by paper ≠ animated interpretation ≠ executed here. |
Course evidence policy |
This is the repository's learning contract, not a K3 paper result. |
thesis · evidence-contract · learning-route |
Animated Interpretation |
Watch the causal picture, inspect retained evidence, recover the derivation, and state the limitation. |
Course learning contract |
The route is an authored pedagogy rather than a paper result. |