EVALUATION

MODULE 2: BENCHMARK DESIGN · SCENE 04

Anatomy of a Benchmark Item

Prompt, reference, scoring rule, metadata, and source. Every field is a design decision that shapes what the score means.

COMPLETENESS · 60%
WHAT THE SCORE MEANS AT THIS COMPLETENESSSCORED BUT UNSLICEABLE — YOU CANNOT DIAGNOSE FAILURES

Drop metadata and you lose diagnosis; drop the scoring rule and you lose the measurement entirely.

ARCHITECTURE & FORMULATIONModule 2: Benchmark Design

Every benchmark item is five design decisions

An item is not just a question. It carries a prompt, a reference answer, a scoring rule, descriptive metadata, and a provenance record. Weakness in any field silently changes what the aggregate score means.

Prompt and reference

The exact prompt template (including few-shot examples) is part of the item; the reference defines what counts as correct.

Scoring rule

Exact match, token F1, rubric, or executable test — chosen before the run, never after seeing results.

Metadata and provenance

Capability leaf, difficulty, source, and license enable slicing and let others reproduce or audit the set.

MATHEMATICAL FORMULATION · THE SCORE IS CONDITIONAL
score = f(item, prompt_template, decoding, scorer)

Change the few-shot count and the same model can move several points. Reporting a bare number without these four inputs is not reproducible science.

REAL-WORLD PRODUCTION ENGINEERING
  • SWE-bench items bundle a repository snapshot, an issue, and a test patch — the scoring rule is executable, not textual.
  • MMLU-Pro added 10-option questions and chain-of-thought prompting to reduce guess rate.
← 03 · Static Suites vs Dynamic Arenas
05 · Difficulty & Discrimination →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]