AI ENGVisual Encyclopedia

MODULE 7: EVALUATION & BENCHMARKING · SCENE 19

Benchmark Contamination & Guardrails

Seen vs. unseen accuracy uplift as the contamination tell, and the guardrails that keep evals honest.

SCANNED · 0%LEAKED ITEMS · 0UPLIFT · +8.2 ± 4.1 PTS
DEDUP SCANNER · ITEMS SEEN IN TRAINING (TOP) vs UNSEEN MATCHES (BOTTOM)BENCHMARK ITEMS SEEN IN TRAINING · 71.2%STRUCTURALLY IDENTICAL UNSEEN ITEMS · 63%UPLIFT GAP: +8.2 PTS · 95% CI ±4.1 PTSGAP EXCEEDS THE CI — CONTAMINATION EVIDENCEGUARDRAIL: DEDUP EVAL ITEMS FROM TRAINING CORPORA BEFORE THE RUN — AND REPORT BOTH SPLITS

RUN to sweep the dedup scanner across the eval bank and compare seen vs unseen accuracy.
GO DEEPER: CONTAMINATION DETECTION IN THE EVALUATION WORLD →

TECHNICAL BREAKDOWNModule 7: Evaluation & Benchmarking

Contamination: the silent score inflator

If benchmark items leaked into training data, the score measures memorization, not capability. Contamination is post-training's special vulnerability: SFT and RLHF datasets are scraped, and popular benchmarks are all over the web.

The Tell

Compare accuracy on seen vs. structurally identical unseen items. A large uplift is contamination evidence; clean models show near-parity.

Guardrails

N-gram and embedding dedup against benchmark corpora before training; hold-out canaries; report both split accuracies when parity cannot be proven.

Rotating Targets

Benchmarks saturate and leak. Private refresh sets, live leaderboards, and newly written items keep the signal alive.

MATHEMATICAL FORMULATION · CONTAMINATION UPLIFT
uplift = acc_seen − acc_unseen, with CI half-width = 1.96·√(p(1−p)/n)

An 8-point uplift with a ±2-point CI is strong evidence; an 8-point uplift with a ±9-point CI is noise. Always report the interval with the gap.

REAL-WORLD PRODUCTION ENGINEERING
  • GSM8K and MATH variants appear verbatim across instruction datasets scraped from GitHub and forums — dedup against eval sets is now table stakes.
  • Report both numbers: 'MATH 62.3 (contamination-checked: uplift 1.1 ± 0.8 vs unseen split)' reads very differently from a bare 62.3.
← 18 · Constitutional AI & AI Critics (RLAIF)
20 · LLM-as-a-Judge: Bias Audit →