Private splits and rotation keep an eval set alive
Every public item is a future training example. A private split — never published, never described in detail — preserves a clean measure of generalization. Rotation periodically replaces a fraction of public items with fresh ones, resetting the ceiling and slowing leakage.
Private split
Held out from papers, leaderboards, and data releases. The only protection against deliberate or accidental training inclusion.
Rotation cadence
Replace a fixed fraction each release window. Larger rotation fights leakage faster but breaks trend comparability.
Version everything
Pin the eval-set version to every reported score so a changed set is visible in the history.
Rotating 25% of items each cycle replaces a quarter of the memorized advantage every release while keeping 75% of the set stable for trend analysis.
- LiveBench refreshes its questions periodically specifically to resist contamination.
- Industrial eval teams maintain an internal private suite that never appears in external reporting.