ARCHITECTURE & FORMULATIONModule 2: Benchmark Design
Design capability coverage before writing items
Aggregate scores are averages over an item mix. If that mix is accidental, one easy, over-represented skill can carry the number. A coverage taxonomy declares the capability leaves first, then budgets items per leaf so the score reflects what you intend to measure.
Declare the leaves
Enumerate the capabilities the suite should cover — arithmetic, retrieval, multi-step reasoning, code, instruction-following — before authoring.
Budget per leaf
Assign item counts deliberately. Re-audit whenever the task or model changes.
Measure concentration
A Herfindahl-style concentration near 1 means the suite is effectively testing one thing.
MATHEMATICAL FORMULATION · COVERAGE CONCENTRATION
H = Σ (items_leaf / items_total)²
Ten leaves with equal counts give H = 0.1. If one leaf holds 70% of items, H ≈ 0.5 — the aggregate is dominated by that one capability.
REAL-WORLD PRODUCTION ENGINEERING
- HELM organizes scenarios and metrics as a coverage grid rather than a single leaderboard number.
- Domain suites (legal, medical) declare a taxonomy of task families and report per-family results.