TECHNICAL BREAKDOWNModule 5: LLM-as-Judge
Audit the judge: position, length, and self-preference
An LLM judge carries systematic biases. It tends to prefer the answer in a particular position, the longer answer, and answers from its own model family. None of these are fatal — but they must be measured and corrected, or the ranking reflects the judge's quirks rather than model quality.
Position bias
Judges often favor the first (or second) slot. Swap the order, re-judge, and average to cancel it.
Length bias
Longer answers win more often even when length is irrelevant. Report length-controlled win rates.
Self-preference
A judge prefers text from its own family. Use a judge from a different family than the candidates.
MATHEMATICAL FORMULATION · ORDER-AVERAGED WIN RATE
win = (win_as_first + win_as_second) / 2
If a model wins 68% in slot A but only 44% in slot B, its true rate is about 56%. Reporting only one order inflates the result by the position bias.
REAL-WORLD PRODUCTION ENGINEERING
- MT-Bench's paper measured GPT-4 position bias and length bias and recommended order-swapping.
- Production judges are validated against a human gold set before their scores are trusted.