Elo turns pairwise votes into a ranking — with uncertainty
Arena evaluation collects pairwise human preferences and updates Elo ratings: an upset moves the winner's rating more than an expected win. Elo is interpretable and robust, but a rating is a distribution, not a fact — the confidence interval and the battle sampling determine what the rank actually means.
Expected score
E_A = 1 / (1 + 10^((R_B − R_A)/400)). The update R_A' = R_A + K(S_A − E_A) moves more on surprise.
Rank is a distribution
Overlapping intervals mean the rank order is not resolved by the data.
Sampling shapes the ranking
Which prompts get battled determines which capabilities the rating reflects. A coding-heavy arena ranks coding ability.
A 1600-rated model against a 1500-rated opponent has expected score 0.64. A win (S=1) with K=32 adds 11.5 points; the same win against a 1800 opponent adds far more.
- Chatbot Arena publishes Elo with bootstrap confidence intervals for every model.
- New models need enough battles before their interval tightens and their rank stabilizes.