EVALUATION

MODULE 6: HUMAN & ARENA EVALUATION · SCENE 18

Arena Elo & Leaderboards

Live battles, Elo updates, and confidence around a rank — why an Elo gap is a distribution, not a fact.

E(A) · 0.500
ANONYMOUS BATTLE · K = 32 · UPSETS MOVE MOREMODEL A · 1500MODEL B · 1500GAP 0 PTS — OVERLAPPING: THE RANK IS UNRESOLVED

Every battle is a vote with a weight. Feed an underdog win and feel the surprise-priced update; then remember a point rating is a distribution.

ARCHITECTURE & FORMULATIONModule 6: Human & Arena Evaluation

Elo turns pairwise votes into a ranking — with uncertainty

Arena evaluation collects pairwise human preferences and updates Elo ratings: an upset moves the winner's rating more than an expected win. Elo is interpretable and robust, but a rating is a distribution, not a fact — the confidence interval and the battle sampling determine what the rank actually means.

Expected score

E_A = 1 / (1 + 10^((R_B − R_A)/400)). The update R_A' = R_A + K(S_A − E_A) moves more on surprise.

Rank is a distribution

Overlapping intervals mean the rank order is not resolved by the data.

Sampling shapes the ranking

Which prompts get battled determines which capabilities the rating reflects. A coding-heavy arena ranks coding ability.

MATHEMATICAL FORMULATION · ELO UPDATE
R_A' = R_A + K · (S_A − E_A)

A 1600-rated model against a 1500-rated opponent has expected score 0.64. A win (S=1) with K=32 adds 11.5 points; the same win against a 1800 opponent adds far more.

REAL-WORLD PRODUCTION ENGINEERING
  • Chatbot Arena publishes Elo with bootstrap confidence intervals for every model.
  • New models need enough battles before their interval tightens and their rank stabilizes.
← 17 · Inter-Annotator Agreement
19 · Confidence Intervals for Scores →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]