Canary strings and trap items make memorization observable
Rather than infer leakage statistically, plant it. Canary strings are unusual tokens embedded in eval documents; if a model can reproduce them, the document was in its training data. Trap items are impossible or self-contradictory tasks whose only correct behavior is to fail — unexpected success means the model is gaming the scorer.
Canary strings
Unique, high-entropy markers (e.g. GUIDs) in eval text. Verbatim recovery is a definitive inclusion test.
Trap items
Impossible questions with no valid answer. A confident wrong answer is expected; a 'correct' one reveals cheating.
Anomaly as signal
Canary hits and trap successes are binary integrity failures, not soft penalties.
MATHEMATICAL FORMULATION · RECOVERY RATE
canary_recovery = hits / canaries_planted
A non-zero recovery rate on canaries that never appeared in a legitimate answer stream proves the model saw the eval text during training.
REAL-WORLD PRODUCTION ENGINEERING
BigBench and private eval providers embed canary GUIDs in their data files for exactly this check.
Code benchmarks use 'impossible' test cases to detect solutions that hard-code expected outputs.