Automated multi-turn self-critique where a model evaluates candidate outputs against written principles and rewires responses.
INTERACTIVE EXHIBIT 10
Constitutional AI: Self-Critique & Revision
ACTIVE PRINCIPLE: 1 / 3
💡 RLAIF: Replaces expensive human preference labeling with scalable, verifiable constitutional self-critique.
ARCHITECTURAL OVERVIEW
Constitutional AI Critique & Revision Loops
Constitutional AI (CAI) automates the alignment process using a written constitution of ethical principles (e.g., helpfulness, harmlessness, non-discrimination). Instead of relying on expensive, noisy human feedback, the model generates an initial response, critiques its own output against constitutional rules, and produces a revised safe completion.
Supervised Learning from AI Feedback (SL-AIF)
During the first phase, a base model is prompted with red-team queries. For each response, a critique prompt asks: 'Critique this response according to Principle X: Is it harmful, toxic, or dangerous?'. The model then rewrites the response based on the critique.
Reinforcement Learning from AI Feedback (RL-AIF)
In the second phase, a specialized preference model is trained on pairs of responses evaluated by AI judges against the constitution, replacing human crowdworkers in the RLHF pipeline.
Transparent & Iterative Policy Evolution
Modifying model behavior requires only updating natural language rules in the constitution, rather than retraining human annotators or re-collecting massive human preference datasets.
MATHEMATICAL FORMULATION · CONSTITUTIONAL PREFERENCE LOSS
Where y_w is the constitutionally revised response and y_l is the unaligned draft. The reward model r_phi learns to assign higher reward to completions that adhere strictly to constitutional constraints.
REAL-WORLD PRODUCTION ENGINEERING
Organize constitutional rules in hierarchical tiers (e.g., Tier 1: Absolute CBRN/Cyber prohibition, Tier 2: Privacy/PII protection, Tier 3: Tone and helpfulness).
Include explicit counter-examples and nuance rubrics in constitutional prompts to prevent excessive over-refusal.
Log full critique-revision audit trails during dataset generation to trace how specific principles altered model behavior.