TECHNICAL BREAKDOWNModule 8: Regression Gates & Monitoring
Turn evaluation into a release gate
An eval that nobody blocks on is documentation. A regression gate attaches a per-capability allowance to the tested axes and fails the build when any axis falls past it. Per-capability budgets matter because a rising average can hide a broken axis — the exact failure a global threshold misses.
Per-capability allowances
Math, code, safety, and instruction-following each get their own tolerance, set from historical noise.
Fail on the axis
Even if the composite improves, a single axis past its allowance blocks the release.
Review the thresholds
Gate thresholds are code: versioned, reviewed, and changed deliberately.
MATHEMATICAL FORMULATION · GATE BREACH
breach if (after − before) < −allowance
A capability allowed to move ±2 points fails when it drops 3. Setting the allowance too tight makes the gate flaky; too loose and real regressions ship.
REAL-WORLD PRODUCTION ENGINEERING
- CI pipelines run a fixed eval subset on every prompt or model change and block the merge on regression.
- Gate allowances are usually derived from the measured run-to-run variance of each suite.