When two models answer the same items, only the items where they disagree carry information about the difference. McNemar's test analyzes those discordant pairs, removing item-difficulty variance and producing a much sharper test than comparing two independent confidence intervals.
The 2×2 table
b = items only A got right, c = items only B got right. Concordant cells (both right, both wrong) add nothing.
McNemar statistic
χ² = (|b − c| − 1)² / (b + c), compared against a chi-square distribution with one degree of freedom.
Sharper by construction
Pairing removes the variance shared by the items, so the same n detects smaller differences.
MATHEMATICAL FORMULATION · MCNEMAR'S TEST
χ² = (|b − c| − 1)² / (b + c)
With b = 30 and c = 10, χ² = 19²/40 = 9.03, comfortably significant at p < 0.01. Comparing two independent 70%–62% intervals on the same items would likely find nothing.
REAL-WORLD PRODUCTION ENGINEERING
Paired bootstrap and McNemar are standard in NLP when comparing two systems on a shared test set.
Models run on identical item sets should always be analyzed with paired methods.