Push both noises to zero and the bar turns fully teal — that is what anchored, single-axis, low-temperature rubrics buy.
ARCHITECTURE & FORMULATIONModule 5: LLM-as-Judge
A judge is only as good as its rubric
When there is no single correct answer, an LLM can grade open-ended output against a rubric. The rubric defines the criteria, the score levels, and observable anchors for each level. Vague rubrics produce noisy, self-inconsistent scores that are impossible to audit.
Criteria and anchors
Define each score level by observable properties ('cites a source', 'arithmetic is correct'), not adjectives like 'good'.
One axis at a time
Scoring helpfulness, correctness, and tone in one pass compounds errors. Split into separate judgments.
Structured output
Force a schema (criterion, score, rationale) so scores can be parsed, aggregated, and audited.
MATHEMATICAL FORMULATION · RUBRIC CONSISTENCY
σ_score² = σ_true² + σ_rubric² + σ_judge²
Observed judge variance mixes real quality differences, rubric ambiguity, and judge sampling. Tightening anchors reduces the middle term; lower temperature reduces the last.
REAL-WORLD PRODUCTION ENGINEERING
MT-Bench uses a structured rubric with a 1–10 scale and explicit criteria for each axis.
Production LLM evaluation often uses binary pass/fail rubrics, which are more reproducible than wide ordinal scales.