EVALUATION

MODULE 5: LLM-AS-JUDGE · SCENE 15

Judge Bias: Position, Length, Self

The referee has preferences. Swap order, control for length, avoid same-family judges — then measure what is left.

WINS AS FIRST ( /100)68WINS AS SECOND ( /100)44
RAW FIRST-SLOT RATE vs ORDER-AVERAGED TRUTHRAW · 0.68 (POSITION BIAS 18 PTS)CORRECTED · 0.56SINGLE-ORDER REPORTING INFLATES BY 12 POINTS — SWAP, RE-JUDGE, AVERAGE

Length and self-preference need the same treatment: control verbosity, and never let a family judge its own.

ARCHITECTURE & FORMULATIONModule 5: LLM-as-Judge

Audit the judge: position, length, and self-preference

An LLM judge carries systematic biases. It tends to prefer the answer in a particular position, the longer answer, and answers from its own model family. None of these are fatal — but they must be measured and corrected, or the ranking reflects the judge's quirks rather than model quality.

Position bias

Judges often favor the first (or second) slot. Swap the order, re-judge, and average to cancel it.

Length bias

Longer answers win more often even when length is irrelevant. Report length-controlled win rates.

Self-preference

A judge prefers text from its own family. Use a judge from a different family than the candidates.

MATHEMATICAL FORMULATION · ORDER-AVERAGED WIN RATE
win = (win_as_first + win_as_second) / 2

If a model wins 68% in slot A but only 44% in slot B, its true rate is about 56%. Reporting only one order inflates the result by the position bias.

REAL-WORLD PRODUCTION ENGINEERING
  • MT-Bench's paper measured GPT-4 position bias and length bias and recommended order-swapping.
  • Production judges are validated against a human gold set before their scores are trusted.
← 14 · Pairwise Comparison & Win Rates
16 · Annotation Protocols & Guidelines →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]