Course 5: Evaluation & Evidence¶
Paper coverage: Section 4.3 (base-model evaluations), Section 5.3 (standard benchmarks), Section 5.4 (real-world tasks) Lab:
docs/evaluation/
What the paper actually measures¶
Base models (Section 4.3)¶
The report frames its base-model protocol explicitly:
"For the evaluation of the base models, we consider benchmarks spanning four key dimensions: world knowledge, language understanding and reasoning, coding and mathematics, and long-context processing." [PAPER Section 4.3.1]
Each dimension has a named suite. World knowledge: AGIEval, C-Eval, CMMLU, MMLU family, MultiLoKo, Simple-QA verified, SuperGPQA, FACTS Parametric, TriviaQA; reasoning: BBH, DROP, HellaSwag, CLUEWSC, WinoGrande; coding/math: BigCodeBench, HumanEval, GSM8K, MATH, MGSM, CMath; long context: LongBench-V2.
Two protocol details deserve attention because they are measurement-hygiene signals:
"All models are evaluated in our internal framework and share the same evaluation setting. Scores with a gap not exceeding 0.3 are considered to be at the same level." [PAPER Table 1 caption]
Declaring a tie-band up front prevents post-hoc winner-picking. Headline results: V4-Flash-Base surpasses V3.2-Base on most benchmarks despite fewer activated parameters; V4-Pro-Base leads across all four dimensions.
Post-trained models (Section 5.3)¶
Knowledge/reasoning move to harder suites (GPQA, Human Last Exam, LiveCodeBench-v6, olympiad math: HMMT, Apex, PutnamBench). The Codeforces methodology is worth quoting because it shows how much care goes into one number:
"For each contest, we generate 32 candidate solutions per problem. For each problem independently, we sample 10 of these solutions without replacement … Each submission is judged against a test suite constructed by domain experts. The score for a solved problem follows the penalty scheme of OpenAI (2025) … The model's overall rating is the average of these contest-level expected ratings across all 14 contests." [PAPER Section 5.3.1]
Formal math runs agentically on Lean v4.28.0-rc1 with up to 500 tool calls; a submission counts only if the strict verifier accepts it. Million-token context is measured on OpenAI MRCR and CorpusQA, re-evaluating Claude Opus 4.6 and Gemini 3.1 Pro "with the goal of standardizing the configuration across all models", and GPT-5.4 is excluded and explained ("its API failed to respond to a large portion of our queries").
Agent capability uses Terminal Bench 2.0, SWE-Verified/Multilingual/Pro, BrowseComp, MCPAtlas, GDPval-AA, Tool-Decathlon under an internal framework providing minimal tools (bash + file edit).
Headline claims, in the authors' own words:
"DeepSeek-V4-Pro-Max … redefines the state-of-the-art for open models." "DeepSeek-V4 series usher in a new era of million-length contexts for open models." [PAPER Abstract, Section 6]
The report also states where it trails: frontier proprietary systems lead the hardest reasoning by roughly 3–6 months, and V4-Pro-Max "slightly worse than frontier closed models" on public agent benchmarks while approaching Claude Opus 4.5 internally.
How to read those tables without fooling yourself¶
Every benchmark number is the last link of a chain: construct → task distribution → interface → metric → model setting → harness → tools → sampling policy → judge → date. Drop any link and the number becomes decoration. The paper models good practice here, declared tie-bands, averaged orderings, verifier-gated correctness, standardized configurations, disclosed API failures.
The honest output of an evaluation is a capability vector, not a universal rank. Accuracy, F1, Elo, pass rates, and pass@k cannot be averaged into one meaningful "overall score." A selected success demonstrates possibility; it never estimates prevalence.
This repo's evaluation contract¶
| tag | what it licenses you to say |
|---|---|
| PAPER | "the report states…" (section cited) |
| MEASURED | "this repo executed… (toy scale)" |
| DERIVED | arithmetic from PAPER facts |
| INFERRED | authored interpretation |
| UNKNOWN | nobody has established it |
Our measured results live entirely at toy scale (CPU, ≤629K params, synthetic max-entropy retrieval tasks). Directions transfer; magnitudes do not. Where our miniatures failed (OPD consolidation economics, sparse-eval generalization of the integrated stack), we documented the failure rather than tuning past it. Failures are evidence too.