Course 5A — Evaluation: Reading Capability Without Fooling Yourself

Section 6 of the Kimi K3 report is not one leaderboard. It is a stack of public benchmarks, internal evaluations, cybersecurity investigations, independent third-party measurements, and cost comparisons. The useful question is therefore not simply “Which model wins?” It is:

What was measured, under which protocol, against which baseline, with which metric, and how far may the resulting claim travel?

The companion visualization teaches the structure and significance of the results. The executed notebook computes selected gaps, paired tool lifts, and Pareto frontiers from transcribed paper values. It does not run K3 or reproduce any benchmark.

The evaluation contract

A benchmark result is better represented as a chain than as a floating number:

\[ \text{construct} \rightarrow \text{tasks} \rightarrow \text{interface} \rightarrow \text{metric} \rightarrow \text{configuration} \rightarrow \text{result} \rightarrow \text{claim}. \]

Source: Author-created evaluation-literacy chain grounded in Kimi K3 §6; not a paper equation.

If any link changes, the meaning may change. An agent score, for example, can depend on the model, reasoning effort, harness, context manager, tool policy, sampling parameters, task revision, judge, and evaluation date.

Three evidence labels are used throughout this course:

Paper reported: the K3 report states the result or method.
Executed analysis: local code performs arithmetic over values transcribed from the paper; it does not rerun the benchmark.
Animated interpretation: a pedagogical model or causal framing supplied by this course.

6.1.1 Benchmarks: four lenses, not one intelligence scalar

The report organizes more than forty public evaluations into four axes:

Axis	What it samples	Representative benchmarks	What it does not establish alone
Reasoning and knowledge	science, legal/critical reasoning, research-level questions	GPQA Diamond, CritPt, AA-LCR, HLE-Full	general reliability on arbitrary research work
Coding	programming, terminal work, real repositories, long-horizon SWE, GPU kernels	ProgramBench, Terminal-Bench 2.1, DeepSWE, FrontierSWE, SWE-Marathon	product-level coding experience under every harness
Agentic	browsing, tools, office work, computer use, finance, legal workflows	BrowseComp, MCPMark, GDPval, OSWorld, Harvey Lab	a harness-independent measure of autonomous agency
Vision	documents, perception, video, charts, mathematics, visual puzzles	OmniDocBench, Video-MME, CharXiv, Math-Vision, ZeroBench	all multimodal perception or embodied behavior

Source: Author-created capability-axis summary of Kimi K3 §6.1.1; not a paper table.

Breadth matters because model quality is not isotropic: one system can be excellent at verified web research and weaker at open-ended scholarly synthesis. Breadth still does not guarantee completeness. Benchmarks overlap, train-test contamination is difficult to rule out, and every suite samples only part of its intended capability.

Representative benchmark cards: definition before significance

These are learner-oriented construct summaries, not claims that each benchmark perfectly measures its intended capability.

Benchmark	What the task asks for	Why it is informative here	Important reading constraint
GPQA Diamond	Answer difficult, expert-written graduate science questions.	Probes compact scientific reasoning and knowledge.	Multiple-choice success is not a research workflow.
HLE-Full	Solve broad frontier academic questions, with separate no-tool and tool-enabled conditions.	Exposes harder research-level gaps and the effect of tools.	Tool and no-tool scores are different systems.
Terminal-Bench 2.1	Complete realistic tasks through a command-line environment.	Tests executable action, debugging, and environment use.	The paper reports the best result across harnesses, so harness selection is part of the score.
SWE-Marathon	Complete extended software-engineering tasks, including calibrated GPU tasks in the paper’s branch.	Samples longer-horizon coding and performance-sensitive work.	K3 uses a pre-release H20-calibrated task branch; cross-paper comparisons require version care.
BrowseComp	Find difficult information through web browsing and synthesis.	Tests search strategy and evidence integration over long trajectories.	K3’s primary result uses context compaction at 300K tokens.
MCPMark-Verified	Use tools exposed through Model Context Protocol tasks with verifiable outcomes.	Separates structured tool execution from fluent claims of completion.	Tool coverage and verifier completeness bound the construct.
OfficeQA Pro	Answer enterprise questions from an entire PDF corpus rendered as images.	Couples document vision, retrieval, and grounded reasoning.	No machine-readable text is supplied under the reported protocol.
OSWorld 2.0	Operate desktop applications to reach a requested state.	Probes computer use, recovery, and long action chains.	Application state and harness behavior can dominate failure modes.
OmniDocBench	Parse structurally diverse PDF documents.	Tests document perception and structured extraction.	Parsing quality is narrower than general visual reasoning.
CharXiv RQ	Answer questions about scientific charts.	Makes Python-assisted inspection and quantitative reading visible.	Tool lift does not reveal which specific tool action caused success.
ZeroBench-main	Solve deliberately difficult visual problems, reported with pass@5.	Stresses perception-plus-reasoning beyond ordinary recognition.	Five attempts and Python access change the compute and interface budget.
Tier-2 cyber suite	Complete end-to-end exploit-development chains in private tasks.	Tests whether discovery becomes verified final execution.	Private targets, refusals, and selective comparison prevent a universal cyber rank.

Source: Benchmark membership and K3 protocol notes come from Kimi K3 §§6.1.1–6.1.3 and §6.2.2. Construct descriptions summarize the cited benchmark purposes; consult each original benchmark before reusing them as formal definitions.

6.1.2 Baselines: preserve the footnotes

The report compares K3 with Claude Fable 5, GPT-5.6 Sol, Claude Opus 4.8, GPT-5.5, and open-weight GLM-5.2. Models are evaluated at maximum reasoning effort except GPT-5.5 at xhigh.

That alignment is useful but not perfect equivalence:

Claude Fable 5 results include fallback behavior.
GPT-5.6 Sol results may include cyberguards.
Some official leaderboard entries bind a model to a particular harness.
Missing table entries mean not reported, not zero.

A fair conclusion keeps these conditions attached to the score rather than treating a model name as the entire experimental setup.

6.1.3 Evaluation configuration is part of the result

For K3, the report states reasoning effort = max and temperature = 1.0. It uses top-p = 0.95 for single-step reasoning and vision tasks without tools and top-p = 1.0 for agentic tasks. The authors generally recommend 0.95 for reasoning/knowledge and 1.0 for coding/agency.

The protocols then diverge by domain:

Coding uses Kimi Code, Claude Code, or Codex. Terminal-Bench reports the best score across harnesses. SWE-Marathon uses an H20-calibrated pre-release task branch; PostTrainBench averages three H20 runs.
BrowseComp uses context compaction at 300K tokens. The report separately gives a 1M-context result without context management.
Vision scores average three runs, except ZeroBench-main with five runs. Several benchmarks are evaluated both with and without Python.
Multiple agentic, reasoning, legal, and finance results are cited from dated third-party leaderboards rather than rerun by the K3 authors.

Compactly:

\[ s = f(M, e, H, T, \theta, D, V, J, t), \]

Source: Author-created notation summarizing protocol factors reported in Kimi K3 §6.1.3; not a paper equation.

where $M$ is the model, $e$ effort, $H$ harness, $T$ tools, $\theta$ sampling, $D$ task distribution, $V$ version, $J$ judge, and $t$ evaluation date.

Metric literacy: do not average unlike numbers

Table 2 mixes several legitimate but incompatible metrics:

Metric	Meaning	Important caveat
Accuracy / pass rate	fraction of items satisfying a criterion	item difficulty and criterion definition dominate transferability
F1	harmonic mean of precision and recall	not directly comparable with accuracy
Elo	relative preference or match rating	depends on opponent pool and drifts with new comparisons
Criterion pass rate	fraction of rubric criteria satisfied	rubric design and judge reliability matter
pass@5	success with up to five attempts	greater sampling budget than pass@1

Source: Author-created metric-literacy summary for reading Kimi K3 Table 2; not a paper table.

An “average K3 score” across these columns would be meaningless. Valid comparisons stay within one row, one metric, and a sufficiently aligned protocol.

6.1.4 Reasoning and knowledge

The profile is mixed rather than contradictory:

Benchmark	K3	Best reported comparator	Gap to best
GPQA Diamond	93.5	GPT-5.6 Sol 94.1	−0.6
AA-LCR	74.7	K3	0.0
CritPt	23.4	GPT-5.6 Sol 32.3	−8.9
HLE-Full, no tools	43.5	Claude Fable 5 53.3	−9.8
HLE-Full, tools	56.0	Claude Fable 5 63.0	−7.0

Source: Values transcribed from Kimi K3 §6.1.4, Table 2, arXiv:2607.24653v2.

Significance. K3 is competitive on graduate science and leads the reported AA-LCR row, but the research-level gap is real in the report. Tool access improves HLE by 12.5 points for K3 without erasing the gap.

Why? The paper itself says research-level reasoning remains a direction for improvement. It does not provide an ablation proving the cause. A plausible hypothesis is that open-ended synthesis supplies a weaker verifier than tasks with executable outcomes, but that is a cross-section interpretation—not a measured causal result.

6.1.4 Coding

Selected contrasts show why one label such as “best coding model” is too coarse:

Benchmark	K3	Fable 5	GPT-5.6 Sol	Reading
ProgramBench	77.8	76.8	77.6	K3 narrowly leads
SWE-Marathon	42.0	35.0	39.0	K3 leads Fable 5 by 7 points
Terminal-Bench 2.1	88.3	88.0	88.8	near tie under best-across-harness reporting
FrontierSWE	81.2	86.6	71.3	K3 second, well ahead of other listed baselines
DeepSWE	67.5	70.0	73.0	K3 trails the top two

Source: Values transcribed from Kimi K3 §6.1.4, Table 2, arXiv:2607.24653v2.

Significance. K3's strongest public evidence includes broad programming and GPU-kernel-oriented SWE. Longer-horizon performance remains benchmark-specific: FrontierSWE is strong, while DeepSWE is not a lead.

Why? The report connects post-training to kernel and agentic task families, making training alignment a plausible contributor. It does not publish a Course-6 ablation that isolates the kernel RL environment as the cause of the seven-point SWE-Marathon lead. Hardware calibration, harness choice, fallbacks, and task version are observed protocol factors and should be discussed before causal training stories.

6.1.4 Agentic evaluations

K3 reports best-in-row results on BrowseComp (91.2), DeepSearchQA (95.0 F1), ResearchRubrics (76.2), MCPMark-Verified (94.5), AutomationBench (30.8), SpreadsheetBench 2 (34.8), $\tau^3$-Banking (33.4), and Harvey Lab-AA (94.6 criterion pass rate).

Important counterexamples remain:

GDPval-AA v2: K3 1,686 Elo; Fable 5 1,747; GPT-5.6 Sol 1,736.
AA-Briefcase: K3 1,548 Elo; Fable 5 1,583.
OSWorld 2.0: K3 58.3; Fable 5 66.1; GPT-5.6 Sol 62.6.
OfficeQA Pro: K3 63.3; Fable 5 69.9.

Significance. The evidence supports strong verified search, tool orchestration, and several professional domains. It does not support “K3 wins agentic tasks” without qualification.

Why? Verifier-aligned training is a plausible explanation for strength on checkable outcomes. Elo-rated suites also introduce judge and opponent-pool effects. Neither observation proves that metric type alone causes the gap: the task distributions, harnesses, and desired response qualities differ simultaneously.

6.1.4 Vision and the Python intervention

The paired no-tool/tool rows are more informative than isolated bars because the same named model is measured under two interfaces:

Benchmark	No Python	With Python	Lift
MMMU-Pro	81.6	83.4	+1.8
CharXiv RQ	84.8	91.3	+6.5
Math-Vision	94.3	97.8	+3.5
ZeroBench-main pass@5	23.0	41.0	+18.0

Source: Values transcribed from the paired Kimi K3 results in §6.1.4, Table 2, arXiv:2607.24653v2.

K3 also leads OmniDocBench at 91.1 and reports 90.0 on Video-MME with subtitles.

Significance. Tool access materially changes the evaluated system, especially on ZeroBench and chart reasoning. The tool lift is not “free model quality”; it measures a model-plus-Python workflow with additional actions and computation.

Why? The intervention supports the narrow claim that Python augmentation helps under these protocols. It does not isolate whether gains come from cropping, OCR, arithmetic, search, multiple attempts, or another behavior without trajectory-level analysis.

The public result is a fingerprint

The report's own overall statement is careful: K3 closely trails Claude Fable 5 and GPT-5.6 Sol while consistently outperforming Claude Opus 4.8, GPT-5.5, and GLM-5.2 across the suite.

It does not publish a universal aggregate over all heterogeneous rows. Consequently, claims such as “third overall” or “fourteen wins” depend on an analyst's counting, treatment of ties, missing values, paired tool variants, and Elo rows. This course uses the paper's domain-level language instead.

6.2.1 Internal evaluation: a failure-finding loop

Kimi's internal suite covers:

coding capability and experience;
general-agent experience;
conversational experience.

The design goal differs from a static public leaderboard. The paper says these suites are refreshed and expanded to track changing failure modes and guide data and training iterations:

\[ \text{observed failures}_t \rightarrow \text{evaluation}_{t+1} \rightarrow \text{data/training}_{t+1}. \]

Source: Author-created feedback-loop notation based on Kimi K3 §6.2.1; not a paper equation.

This can improve product relevance and reduce overfitting to stale public tests. It also reduces auditability because external learners do not receive the private tasks, rubrics, raw trajectories, or judge calibration.

Internal results: sharper strengths and weaknesses

The paper highlights orchestration and research strengths: K3 leads Swarm Bench at 76.3 and Deep Research Bench at 90.0. It trails only Fable 5 on Kimi Code Bench 2.0 under Claude Code, leads the reported Coding Experience row, and is preferred over Opus 4.8 by a +31.0 win-minus-lose margin on Kimi Webdev Bench.

The authors also identify gaps on Agent Behavior Bench, MIRA Bench, 24/7 ClawBench 2.0, Agentic Vision Bench, and KWV Bench. This is pedagogically important: an internal evaluation program is credible when it exposes uncomfortable product-specific weaknesses, not only wins.

Footnotes still matter. Harness assignments differ, several baselines refuse some tasks, and some cells remain unreported.

6.2.2 Cybersecurity: evaluate capability in tiers

The report separates:

Tier 1 — vulnerability discovery and proof of concept: primarily defensive discovery in current codebases.
Tier 2 — end-to-end exploit development: operationally higher risk because the system must complete the exploit chain.

For reviewed Tier-1 candidates, the report states that approximately 70% were confirmed genuine, including 16 previously unknown vulnerabilities across six projects. The denominator is findings that received human review, not every generated candidate.

For the private 36-task Tier-2 suite:

Model	Solved	Rate
K3	14 / 36	38.9%
GLM-5.2	8 / 36	22.2%

Source: Kimi K3 §6.2.2 private Tier-2 cyber evaluation, arXiv:2607.24653v2.

Ten of K3's fourteen successes come from the sixteen-task user-space track. Neither model solves three quarters of the twenty kernel tasks. Proprietary frontier models are excluded because refusals make comparison infeasible.

This is capability evidence and risk-relevant measurement, not an instruction to perform exploitation.

Cyber gap: the paper supplies trajectory-level causes

Unlike broad speculation about benchmark gaps, the report attributes failures from trajectory analysis to four recurring patterns:

failing to finish an exploit chain after obtaining useful primitives;
poor strategy selection under mitigations;
prolonged unproductive debugging loops;
insufficient verification before submission.

An independent UK AISI–NIST CAISI assessment is directionally consistent: K3 beats GLM-5.2 on reported exploit-development measures but trails frontier cyber-capable models on end-to-end completion, including zero arbitrary-code-execution successes on 41 tasks. The paper explicitly calls its own evaluation a lower bound conditioned on model version and evaluation coverage.

6.3 Third-party evaluation: triangulate, do not canonize

As reported on July 23, 2026:

External lens	K3 result	Reported rank	What it emphasizes
Artificial Analysis Intelligence Index v4.1	57.1	4 / 580	composite benchmark index
Vals Index	74.7%	2 / 39	GDP-weighted industry tasks
WebDev Arena	1,678 Elo	1 / 99	crowdsourced web-development preference
Text Arena	1,486 Elo	8 / 200	crowdsourced text preference
Agent Arena	9.1	4 / 37	emerging agent preference measure

Source: Third-party snapshot values as reported by Kimi K3 §6.3, Table 5, on July 23, 2026.

The disagreement is the lesson. K3 can rank first for web development and eighth for general text at the same snapshot because these evaluations sample different constructs. Independent measurement reduces dependence on the developer's own suite; it does not make a leaderboard timeless or methodologically perfect.

6.4 Cost efficiency and the Pareto frontier

Capability per dollar is not simply $\text{score}/\text{cost}$ because ratios can behave badly across arbitrary score origins. A cleaner concept is Pareto dominance. Model $j$ dominates model $i$ if:

\[ c_j \le c_i \quad\land\quad s_j \ge s_i, \]

Source: Standard Pareto-dominance definition used to interpret Kimi K3 §6.4 and Fig. 13; not a paper-specific result.

with at least one strict inequality. A point is on the frontier if no other point dominates it.

The report compares score and per-task inference cost on Kimi Code Bench 2.0, BrowseComp, GDPval-AA v2, and AA-Briefcase. It reports:

K3 is four KCB points behind Fable 5 at 38% of its cost.
K3 scores 91.2 on BrowseComp at $2.03 per task, half the GPT-5.6 Sol cost and roughly an order of magnitude below maximum-effort Claude models.
K3 is within 50 GDPval Elo of GPT-5.6 Sol at 13% lower cost and 2.6× cheaper than Fable 5.
K3 is second on AA-Briefcase at roughly half Fable 5's cost.

Boundary. The cost sources differ: internal measurements, published provider charts, and Artificial Analysis API pricing. Harness overhead, token prices, caching, and effort settings can move the frontier.

Explaining “why” without outrunning the evidence

Use a four-rung evidence ladder:

Rung	Example	Allowed language
Observed result	K3 gains 18 points on ZeroBench with Python	“The reported score increases under Python augmentation.”
Observed protocol factor	Terminal-Bench takes the best score across harnesses	“Harness selection is part of this result.”
Paper-supported causal diagnosis	cyber traces repeatedly stall in debugging loops	“The paper attributes failures to…”
Plausible hypothesis	verifier-rich RL may favor checkable task families	“A plausible contributor is…”

Source: Author-created evidence ladder for interpreting Kimi K3 §§4 and 6; not a paper table.

Do not promote the fourth rung to the third. In particular:

a benchmark lead does not prove which training component caused it;
correlation between verifiability and strength is not an ablation;
a tool lift does not identify the successful tool behavior;
an Elo gap may involve task quality, judge preference, harness behavior, or model capability simultaneously.

The right goal is not to avoid explanations. It is to make their evidence grade visible.

Synthesis

Course 5A supports the following bounded reading:

K3 is an open-weight model close to the strongest proprietary systems in the paper's broad suite.
Its public profile is especially strong in several coding, verified search/tool, document, and professional-domain tasks.
Research-level reasoning, some knowledge-work preferences, harder computer use, and several internal behavior/coordination suites expose real gaps.
Python augmentation produces meaningful paired improvements on several vision benchmarks.
Internal and cybersecurity evaluations add failure-mode detail unavailable from public rankings, but come with stronger auditability constraints.
Independent rankings triangulate different aspects of quality rather than collapsing into one stable rank.
K3 is reported on or near the cost-efficiency frontier for four studied workloads.

The discipline is simple:

Measure precisely. Compare locally. Triangulate independently. Explain cautiously. Stop where the evidence stops.

Retrieval check

Choose one benchmark result and name its metric, protocol qualification, provenance, and one causal explanation the score does not establish. Then choose one cost-frontier point and explain why “cheapest” and “Pareto-efficient” are different claims.