AI ENGVisual Encyclopedia

MODULE 2: BENCHMARK DESIGN · SCENE 06

Capability Coverage & Taxonomy

Map items to capability leaves before writing them. Balanced coverage prevents a single easy skill from carrying the score.

ARITHMETIC10RETRIEVAL10REASONING10CODE10H = 0.25
ITEM MIX · 25 / 25 / 25 / 25%H = 0.25 — BALANCED COVERAGE, THE MEAN EARNS ITS MEANING

Starve a leaf and failures there vanish from the average. Budget coverage before authoring items, then re-audit.

TECHNICAL BREAKDOWNModule 2: Benchmark Design

Design capability coverage before writing items

Aggregate scores are averages over an item mix. If that mix is accidental, one easy, over-represented skill can carry the number. A coverage taxonomy declares the capability leaves first, then budgets items per leaf so the score reflects what you intend to measure.

Declare the leaves

Enumerate the capabilities the suite should cover — arithmetic, retrieval, multi-step reasoning, code, instruction-following — before authoring.

Budget per leaf

Assign item counts deliberately. Re-audit whenever the task or model changes.

Measure concentration

A Herfindahl-style concentration near 1 means the suite is effectively testing one thing.

MATHEMATICAL FORMULATION · COVERAGE CONCENTRATION
H = Σ (items_leaf / items_total)²

Ten leaves with equal counts give H = 0.1. If one leaf holds 70% of items, H ≈ 0.5 — the aggregate is dominated by that one capability.

REAL-WORLD PRODUCTION ENGINEERING
  • HELM organizes scenarios and metrics as a coverage grid rather than a single leaderboard number.
  • Domain suites (legal, medical) declare a taxonomy of task families and report per-family results.
← 05 · Difficulty & Discrimination
07 · Contamination Detection →