EVALUATION

MODULE 1: EVAL FOUNDATIONS · SCENE 03

Static Suites vs Dynamic Arenas

Frozen item banks are reproducible but leak; live arenas resist contamination but drift with the crowd. Serious reports carry both.

ROTATION / RELEASE25%RELEASE ROUNDS4
CONTAMINATION RISKSTATIC SUITE · 32%LIVE ARENA · 8%ROTATION KEEPS THE STATIC SIGNAL HONEST · STILL CARRY BOTH

Static suites are comparable until they leak; arenas stay fresh but their population drifts. Reproducibility and freshness are different goods.

ARCHITECTURE & FORMULATIONModule 1: Eval Foundations

Static suites and dynamic arenas trade off reproducibility for freshness

Frozen item banks are the backbone of reproducible comparison — but every public set eventually leaks into training data. Live arenas compare models on fresh traffic and resist contamination, at the cost of a moving population and a harder-to-audit protocol.

Static: comparable, leaky

Same items, same order, same scoring across releases. The longer a set is public, the more its items appear in pre-training corpora.

Dynamic: fresh, drifting

Arena-style battles sample real prompts continually, but the user population and prompt mix shift over time.

Carry both

Use static suites for regression and trend, arenas for contamination-resistant ranking. Label each number's provenance.

MATHEMATICAL FORMULATION · LEAKAGE RISK OVER ROUNDS
risk = static_share · (1 − rotation)^rounds + (1 − static_share) · 0.05

With 100% static items and no rotation, risk stays at the floor imposed by leakage. Rotating 30% of items per release drops the static contribution by 0.7 each round.

REAL-WORLD PRODUCTION ENGINEERING
  • LMSYS Chatbot Arena pairs anonymous models on user prompts and ranks them with Elo/Bradley-Terry from live votes.
  • HELM and MMLU-Pro are static and versioned; leaderboards pin the version alongside the score.
← 02 · Capability vs Alignment Evals
04 · Anatomy of a Benchmark Item →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]