EVALUATION

MODULE 7: STATISTICAL RIGOR · SCENE 19

Confidence Intervals for Scores

Every accuracy is a draw from a sampling distribution. Report the interval and the n that produced it, or the score is a rumor.

ITEMS nn = 500h ≈ ±0.040
0.60.70.8A 0.70 ± 0.040B 0.74 ± 0.038INTERVALS OVERLAP — 70 vs 74 ON THIS n IS A RUMOR, NOT A RESULT

Drag n upward and watch the whiskers tighten as 1/√n. Four times the items buys half the uncertainty — precision is expensive.

ARCHITECTURE & FORMULATIONModule 7: Statistical Rigor

Every score is a draw from a sampling distribution

A benchmark score is computed on one sample of items. Had the set been drawn differently, the score would differ. The confidence interval quantifies that variability. Reporting a point estimate without n and an interval is the most common statistical error in model evaluation.

Standard error

For a proportion, SE = √(p(1−p)/n). It shrinks as 1/√n, so precision gets expensive.

95% interval

h = 1.96·SE. For 500 items at 70% accuracy, h ≈ ±4.0 points.

Compare intervals

Claim an improvement only when the intervals separate — or better, use a paired test.

MATHEMATICAL FORMULATION · HALF-WIDTH AND SAMPLE SIZE
h = 1.96·√(p(1−p)/n), n = (1.96/h)²·p(1−p)

Halving the interval requires four times the items. To resolve a 2-point difference at 50% accuracy you need roughly 9,600 items per model.

REAL-WORLD PRODUCTION ENGINEERING
  • HELM reports standard errors alongside every aggregate metric.
  • Leaderboards that rank models within a point or two without intervals are reporting noise.
← 18 · Arena Elo & Leaderboards
20 · Paired Tests & McNemar →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]