Constitutional AI: AI critics judging AI actors
RLAIF replaces human preference labelers with an AI judge that grades responses against a written constitution. Constitutional AI combines that with self-critique: the model critiques its own draft, revises, and the revised responses become preference data.
Critique → Revise
Generate a response, ask the model to critique it against a principle, then rewrite. The revised pair (original vs revised) is a training example with no human in the loop.
The Constitution
A set of principles (harmless, honest, non-manipulative...) written in natural language. Editing the constitution edits behavior — governance becomes a text file.
Judge Calibration
AI judges agree with human panels ~70–75% baseline; agreement improves with judge scale and better rubrics but never reaches parity on edge cases.
Each constitutional revision pass cuts the violation rate roughly in half, while judge reliability grows only logarithmically in judge scale — the asymmetry that keeps humans in the loop.
- Anthropic's CAI showed the full loop: harmless and helpful both improve with AI feedback when the constitution is clear and the judge is prompted for pairwise comparison.
- Operational risk: the judge inherits the base model's biases. Audit RLAIF labels against a human holdout set before trusting the loop.