Resample for intervals, then pay for every comparison
The bootstrap builds a sampling distribution by resampling the observed data, without assuming normality — useful for medians, ratios, and Elo. But testing many slices at once inflates false positives. The multiple-comparisons tax (Bonferroni, Holm) must be paid on every additional hypothesis.
Bootstrap
Draw n items with replacement many times, recompute the statistic each time, and read percentiles as the interval.
Family-wise error
Twenty independent tests at α = 0.05 expect one false positive by chance alone.
Holm correction
Sort p-values and compare each to α/(m − i + 1), stopping at the first failure. Uniformly more powerful than Bonferroni.
MATHEMATICAL FORMULATION · HOLM STEP-DOWN
reject p_(i) if p_(i) ≤ α / (m − i + 1)
With p-values [0.001, 0.01, 0.04, 0.20] and α = 0.05: the first two fall below 0.0125 and 0.0167, the third (0.04) does not clear 0.025, so three comparisons are rejected in sequence — exactly two.
REAL-WORLD PRODUCTION ENGINEERING
Arena Elo intervals are frequently computed with the bootstrap over battles.
Benchmark papers comparing many models across many tasks must correct for multiplicity or report it explicitly.