A judge is only as good as its rubric
When there is no single correct answer, an LLM can grade open-ended output against a rubric. The rubric defines the criteria, the score levels, and observable anchors for each level. Vague rubrics produce noisy, self-inconsistent scores that are impossible to audit.
Criteria and anchors
Define each score level by observable properties ('cites a source', 'arithmetic is correct'), not adjectives like 'good'.
One axis at a time
Scoring helpfulness, correctness, and tone in one pass compounds errors. Split into separate judgments.
Structured output
Force a schema (criterion, score, rationale) so scores can be parsed, aggregated, and audited.
Observed judge variance mixes real quality differences, rubric ambiguity, and judge sampling. Tightening anchors reduces the middle term; lower temperature reduces the last.
- MT-Bench uses a structured rubric with a 1–10 scale and explicit criteria for each axis.
- Production LLM evaluation often uses binary pass/fail rubrics, which are more reproducible than wide ordinal scales.