Training refusals without breaking helpfulness: the quadratic cost of over-refusal on the frontier curve.
REFUSAL WEIGHTREFUSALS · 40%HELPFULNESS · 94.4%
RUN to send a stream of harmful and benign near-boundary probes at the current calibration.
ARCHITECTURE & FORMULATIONModule 6: Safety & Constitutional AI
Refusal calibration and the over-refusal trap
A model that refuses everything is safe and useless. Refusal training walks a frontier: enough refusals on genuinely harmful inputs, without the quadratic helpfulness tax of refusing benign look-alikes (medical questions, historical discussion, security research).
The Frontier
Each safety-training intensity yields a point (refusal_rate, helpfulness). Over-refusal cost grows quadratically because false refusals compound across a conversation.
Boundary Data
Curate near-boundary pairs: same surface form, one harmful one benign. The model must learn the decision variable, not the surface pattern.
Calibration Check
Evalidate on both a harmful-prompt suite (want: high refusal) and an over-refusal suite like XSTest (want: low refusal). One number hides the failure.
With safety training s=0.8 and refusal weight w=0.5: refusals 0.40, helpfulness 0.944. Pushing w to 1.0 gives refusals 0.80 but helpfulness drops to 0.776 — the trap made visible.
REAL-WORLD PRODUCTION ENGINEERING
Post-launch telemetry: track false-refusal complaints separately from true-positive refusals; regressions show up as a helpfulness dip on benign clusters.
Constitutional or rubric-based synthetic boundary data lets you generate near-boundary pairs at scale without hand-writing thousands.