EVALUATION

MODULE 2: BENCHMARK DESIGN · SCENE 05

Difficulty & Discrimination

Item difficulty p and discrimination d: why floor and ceiling items waste budget and what a useful spread looks like.

DIFFICULTY BANDp ∈ [0.10, 0.90]5/6 KEPT · MEAN d 0.51
multi-digit multiply · p 0.12 · d 0.68code repair · p 0.38 · d 0.61logic puzzle · p 0.47 · d 0.58commonsense QA · p 0.55 · d 0.45news summary · p 0.81 · d 0.22capital recall · p 0.94 · d 0.08 · CULLEDSIZE = DISCRIMINATION · BAND EDGES = CULLING THRESHOLDS

Floor and ceiling items look easy to write and cost you information. Keeping everything keeps the noise too.

ARCHITECTURE & FORMULATIONModule 2: Benchmark Design

Difficulty and discrimination decide an item's value

Two classical item statistics tell you whether an item earns its place. Difficulty p is the fraction answering correctly. Discrimination d is how sharply the item separates stronger from weaker models. Items at the floor or ceiling add noise without information.

Difficulty p

p near 0.0 is a floor item (everyone fails), p near 1.0 a ceiling item (everyone passes). Both provide little signal.

Discrimination d

d = p(top group) − p(bottom group). High d means the item tracks the capability under test.

Aim for a spread

A test that is uniformly hard produces a compressed score distribution and a wide confidence interval.

MATHEMATICAL FORMULATION · DISCRIMINATION INDEX
d = p_top − p_bottom

If the top quartile of models answers 0.9 and the bottom quartile 0.2, d = 0.7 — a strongly discriminating item. If both answer 0.5, d = 0 and the item is uninformative.

REAL-WORLD PRODUCTION ENGINEERING
  • Psychometrics teams routinely cull items with p < 0.1 or p > 0.9 before freezing a suite.
  • Adaptive testing picks the next item near the examinee's current ability to maximize information per item.
← 04 · Anatomy of a Benchmark Item
06 · Capability Coverage & Taxonomy →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]