ARCHITECTURE & FORMULATIONModule 7: Evaluation & Benchmarking
Honest reporting: confidence intervals on every score
A benchmark score without an interval is a point estimate pretending to be a fact. On a 500-item benchmark, a 70% score carries a ±4-point 95% CI — enough to make most adjacent-leaderboard comparisons meaningless.
Normal Approximation
CI half-width = 1.96·√(p(1−p)/n). It shrinks with √n: doubling items improves precision by only 41%.
Overlapping Intervals
Model A at 71.2 ± 1.5 vs Model B at 69.8 ± 1.6: the difference is inside the noise. Claim improvement only when intervals separate.
Paired Tests
McNemar's test on the same items is sharper than comparing two independent CIs — items both models get right carry no information about the difference.
MATHEMATICAL FORMULATION · CI HALF-WIDTH
h = 1.96 · √( p(1−p) / n ), e.g. p=0.7, n=500 → h ≈ ±0.040
70% on 500 items is 70% ± 4.0 points at 95% confidence. Beating a rival by 1.5 points on that benchmark is not evidence of anything.
REAL-WORLD PRODUCTION ENGINEERING
- Publish n and the interval next to every headline number — the HELM and lm-eval-harness conventions make this near-default in serious reports.
- For pass@k and small-sample reasoning evals, use bootstrap CIs; the normal approximation is poor near 0/1 accuracy.