POST-TRAINING & DISTILLATION

MODULE 6: SAFETY & CONSTITUTIONAL AI · SCENE 18

Constitutional AI & AI Critics (RLAIF)

Self-critique, revision loops, and judge agreement with human panels as the violation rate decays.

REVISION PASS · 0VIOLATIONS · 42.0%JUDGE↔HUMAN · 92%
1. DRAFT→2. CRITIQUE→3. REVISE→4. PREFER42%25%15%9%5%3%

The model critiques its own draft against a written constitution, revises, and the pair becomes preference data.

ARCHITECTURE & FORMULATIONModule 6: Safety & Constitutional AI

Constitutional AI: AI critics judging AI actors

RLAIF replaces human preference labelers with an AI judge that grades responses against a written constitution. Constitutional AI combines that with self-critique: the model critiques its own draft, revises, and the revised responses become preference data.

Critique → Revise

Generate a response, ask the model to critique it against a principle, then rewrite. The revised pair (original vs revised) is a training example with no human in the loop.

The Constitution

A set of principles (harmless, honest, non-manipulative...) written in natural language. Editing the constitution edits behavior — governance becomes a text file.

Judge Calibration

AI judges agree with human panels ~70–75% baseline; agreement improves with judge scale and better rubrics but never reaches parity on edge cases.

MATHEMATICAL FORMULATION · VIOLATION DECAY & JUDGE AGREEMENT
violations(p) = 0.42 × 0.6^p; agreement(n) = 0.72 + 0.2·log₁₀(max(10,n)/1000) (capped 0.95)

Each constitutional revision pass cuts the violation rate roughly in half, while judge reliability grows only logarithmically in judge scale — the asymmetry that keeps humans in the loop.

REAL-WORLD PRODUCTION ENGINEERING
  • Anthropic's CAI showed the full loop: harmless and helpful both improve with AI feedback when the constitution is clear and the judge is prompted for pairwise comparison.
  • Operational risk: the judge inherits the base model's biases. Audit RLAIF labels against a human holdout set before trusting the loop.
← 17 · Refusal Calibration & the Over-Refusal Trap
19 · Benchmark Contamination & Guardrails →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]