LLM-as-a-judge: auditing the referee
LLM judges scale preference evaluation to thousands of comparisons, but they carry measurable biases: position (preferring the first answer), length (preferring the longer), and self-preference. An unaudited judge corrupts every win-rate downstream.
Positional Bias
Run every comparison twice with answers swapped. Win-rate flips reveal order dependence; report the bias-corrected average.
Length Bias
Verbose answers win more than their quality warrants. Length-controlled win rates (AlpacaEval-LC) strip the confound.
Judge Elo
Pairwise win rates convert to Elo via a Bradley-Terry fit — the same model as Module 4, now scoring models instead of responses.
A judge that prefers the first answer 70% of the time carries bias 0.2 — enough to flip close comparisons. A 64% win rate converts to +128 Elo; a 55% rate, only +40.
- MT-Bench and Chatbot Arena popularized the pattern: judge rubric + swap-augmented pairs + length control.
- Self-preference is real: a judge from the same model family rates its own outputs higher. Use a judge from a different family, or an ensemble.