Static suites and dynamic arenas trade off reproducibility for freshness
Frozen item banks are the backbone of reproducible comparison — but every public set eventually leaks into training data. Live arenas compare models on fresh traffic and resist contamination, at the cost of a moving population and a harder-to-audit protocol.
Static: comparable, leaky
Same items, same order, same scoring across releases. The longer a set is public, the more its items appear in pre-training corpora.
Dynamic: fresh, drifting
Arena-style battles sample real prompts continually, but the user population and prompt mix shift over time.
Carry both
Use static suites for regression and trend, arenas for contamination-resistant ranking. Label each number's provenance.
With 100% static items and no rotation, risk stays at the floor imposed by leakage. Rotating 30% of items per release drops the static contribution by 0.7 each round.
- LMSYS Chatbot Arena pairs anonymous models on user prompts and ranks them with Elo/Bradley-Terry from live votes.
- HELM and MMLU-Pro are static and versioned; leaderboards pin the version alongside the score.