TECHNICAL BREAKDOWNModule 7: Statistical Rigor
Every score is a draw from a sampling distribution
A benchmark score is computed on one sample of items. Had the set been drawn differently, the score would differ. The confidence interval quantifies that variability. Reporting a point estimate without n and an interval is the most common statistical error in model evaluation.
Standard error
For a proportion, SE = √(p(1−p)/n). It shrinks as 1/√n, so precision gets expensive.
95% interval
h = 1.96·SE. For 500 items at 70% accuracy, h ≈ ±4.0 points.
Compare intervals
Claim an improvement only when the intervals separate — or better, use a paired test.
MATHEMATICAL FORMULATION · HALF-WIDTH AND SAMPLE SIZE
h = 1.96·√(p(1−p)/n), n = (1.96/h)²·p(1−p)
Halving the interval requires four times the items. To resolve a 2-point difference at 50% accuracy you need roughly 9,600 items per model.
REAL-WORLD PRODUCTION ENGINEERING
- HELM reports standard errors alongside every aggregate metric.
- Leaderboards that rank models within a point or two without intervals are reporting noise.