AI ENGVisual Encyclopedia

MODULE 5: LLM-AS-JUDGE · SCENE 14

Pairwise Comparison & Win Rates

Compare two answers instead of scoring one. Lower variance, harder aggregation, and the Bradley-Terry bridge to Elo.

RATING GAP (A − B)+150 EloP(A ≻ B) · 0.703
MODEL A vs MODEL B · TIES ALLOWED IN THE PROTOCOLA WINS · 70.3%B WINS · 29.7%WIN RATE 0.703 ↔ GAP 150 ELO — THE BRIDGE RUNS BOTH WAYS

Relative judgment is easy; ranking is the hard part. Bradley-Terry turns scattered A-vs-B outcomes into one shared scale.

TECHNICAL BREAKDOWNModule 5: LLM-as-Judge

Pairwise comparison buys lower variance at an aggregation cost

Humans and judges are far more consistent at choosing between two answers than at assigning absolute scores. Pairwise comparison asks 'which is better?' and allows ties. The cost is that win rates must be aggregated across pairs to recover a ranking, usually through Bradley-Terry or Elo.

Lower variance

Relative judgments remove scale ambiguity: the judge only needs to order two options, not calibrate an absolute scale.

Allow ties

Forcing a winner when answers are equivalent injects false wins and biases the ranking.

Aggregate to a ranking

Fit Bradley-Terry: P(A beats B) = σ(r_A − r_B), then convert ratings to win rates.

MATHEMATICAL FORMULATION · BRADLEY-TERRY
P(A ≻ B) = 1 / (1 + e^−(r_A − r_B))

A rating gap of 2.2 logits corresponds to roughly a 90% win probability. Pairwise outcomes across many models fit this single model jointly.

REAL-WORLD PRODUCTION ENGINEERING
  • Chatbot Arena exposes only pairwise battles and fits a Bradley-Terry model to produce Elo-like ratings.
  • Pairwise judging requires O(n²) comparisons for n models, so organizers use sparse battle schedules.
← 13 · Rubric Design & Pointwise Scoring
15 · Judge Bias: Position, Length, Self →