TECHNICAL BREAKDOWNModule 3: Contamination & Integrity
Detecting leakage between eval items and training data
If an eval item appeared in pre-training, its score measures memorization, not generalization. Contamination checks compare eval items against the training corpus at the n-gram and embedding level, then rank items by overlap so leaked items can be deduplicated or dropped.
n-gram overlap
Fraction of an item's n-grams also present in the corpus. High overlap on long spans is strong evidence of inclusion.
Perplexity tells
Anomalously low loss on an item the model should find hard is a memorization signature.
Report the residual
After cleaning, publish how much overlap remains rather than claiming a clean set.
MATHEMATICAL FORMULATION · OVERLAP RATIO
overlap = |ngrams(item) ∩ ngrams(corpus)| / |ngrams(item)|
With n = 13, even a modest overlap indicates a shared passage. Systems like GPT-3's report used n-gram overlap to flag benchmark contamination.
REAL-WORLD PRODUCTION ENGINEERING
- The GPT-3 paper used 13-gram overlap against Common Crawl to estimate benchmark contamination.
- Modern suites ship a contamination report and often a held-out private split.