Paired tests compare models on the same items
When two models answer the same items, only the items where they disagree carry information about the difference. McNemar's test analyzes those discordant pairs, removing item-difficulty variance and producing a much sharper test than comparing two independent confidence intervals.
The 2×2 table
b = items only A got right, c = items only B got right. Concordant cells (both right, both wrong) add nothing.
McNemar statistic
χ² = (|b − c| − 1)² / (b + c), compared against a chi-square distribution with one degree of freedom.
Sharper by construction
Pairing removes the variance shared by the items, so the same n detects smaller differences.
With b = 30 and c = 10, χ² = 19²/40 = 9.03, comfortably significant at p < 0.01. Comparing two independent 70%–62% intervals on the same items would likely find nothing.
- Paired bootstrap and McNemar are standard in NLP when comparing two systems on a shared test set.
- Models run on identical item sets should always be analyzed with paired methods.