TECHNICAL BREAKDOWNModule 2: Benchmark Design
Every benchmark item is five design decisions
An item is not just a question. It carries a prompt, a reference answer, a scoring rule, descriptive metadata, and a provenance record. Weakness in any field silently changes what the aggregate score means.
Prompt and reference
The exact prompt template (including few-shot examples) is part of the item; the reference defines what counts as correct.
Scoring rule
Exact match, token F1, rubric, or executable test — chosen before the run, never after seeing results.
Metadata and provenance
Capability leaf, difficulty, source, and license enable slicing and let others reproduce or audit the set.
MATHEMATICAL FORMULATION · THE SCORE IS CONDITIONAL
score = f(item, prompt_template, decoding, scorer)
Change the few-shot count and the same model can move several points. Reporting a bare number without these four inputs is not reproducible science.
REAL-WORLD PRODUCTION ENGINEERING
- SWE-bench items bundle a repository snapshot, an issue, and a test patch — the scoring rule is executable, not textual.
- MMLU-Pro added 10-option questions and chain-of-thought prompting to reduce guess rate.