EVALUATION

MODULE 3: CONTAMINATION & INTEGRITY · SCENE 07

Contamination Detection

n-gram overlap, embedding proximity, and perplexity tells that reveal whether eval items leaked into training data.

N-GRAM SIZEn = 5
MAX OVERLAP vs TRAINING CORPUSverbatim passage · 100% LEAKEDparaphrase · 0% CLEANunrelated · 0% CLEAN

Seen accuracy 0.712 vs unseen 0.63: an uplift of 0.082 is the contamination tell. Larger n sharpens the verdict.

ARCHITECTURE & FORMULATIONModule 3: Contamination & Integrity

Detecting leakage between eval items and training data

If an eval item appeared in pre-training, its score measures memorization, not generalization. Contamination checks compare eval items against the training corpus at the n-gram and embedding level, then rank items by overlap so leaked items can be deduplicated or dropped.

n-gram overlap

Fraction of an item's n-grams also present in the corpus. High overlap on long spans is strong evidence of inclusion.

Perplexity tells

Anomalously low loss on an item the model should find hard is a memorization signature.

Report the residual

After cleaning, publish how much overlap remains rather than claiming a clean set.

MATHEMATICAL FORMULATION · OVERLAP RATIO
overlap = |ngrams(item) ∩ ngrams(corpus)| / |ngrams(item)|

With n = 13, even a modest overlap indicates a shared passage. Systems like GPT-3's report used n-gram overlap to flag benchmark contamination.

REAL-WORLD PRODUCTION ENGINEERING
  • The GPT-3 paper used 13-gram overlap against Common Crawl to estimate benchmark contamination.
  • Modern suites ship a contamination report and often a held-out private split.
← 06 · Capability Coverage & Taxonomy
08 · Canary Strings & Trap Items →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]