If benchmark items leaked into training data, the score measures memorization, not capability. Contamination is post-training's special vulnerability: SFT and RLHF datasets are scraped, and popular benchmarks are all over the web.
The Tell
Compare accuracy on seen vs. structurally identical unseen items. A large uplift is contamination evidence; clean models show near-parity.
Guardrails
N-gram and embedding dedup against benchmark corpora before training; hold-out canaries; report both split accuracies when parity cannot be proven.
Rotating Targets
Benchmarks saturate and leak. Private refresh sets, live leaderboards, and newly written items keep the signal alive.
MATHEMATICAL FORMULATION · CONTAMINATION UPLIFT
uplift = acc_seen − acc_unseen, with CI half-width = 1.96·√(p(1−p)/n)
An 8-point uplift with a ±2-point CI is strong evidence; an 8-point uplift with a ±9-point CI is noise. Always report the interval with the gap.
REAL-WORLD PRODUCTION ENGINEERING
GSM8K and MATH variants appear verbatim across instruction datasets scraped from GitHub and forums — dedup against eval sets is now table stakes.
Report both numbers: 'MATH 62.3 (contamination-checked: uplift 1.1 ± 0.8 vs unseen split)' reads very differently from a bare 62.3.