TECHNICAL BREAKDOWNModule 8: Regression Gates & Monitoring
A frozen golden set catches quiet breakage
Prompt changes are code changes with no compiler. Editing one instruction to fix an edge case can silently break fifty other cases. A golden set — a frozen, versioned collection of representative inputs and expected behaviors — is re-run on every change and diffed item by item.
Freeze and version
The golden set is immutable per version. New cases are added in a new version so old comparisons stay valid.
Diff, don't average
Aggregate scores hide swaps: two items improving can mask two breaking. Inspect per-item flips.
Tie prompt to evidence
Every prompt version records the eval run that approved it, so regressions can be bisected.
MATHEMATICAL FORMULATION · ADVERSE REGRESSION RATE
rate = regressed_items / tested_items
If 40 of 2,000 golden items flip from pass to fail, the adverse regression rate is 2% — small in aggregate, but decisive if those items cover a critical capability.
REAL-WORLD PRODUCTION ENGINEERING
- Prompt-management systems store prompt versions alongside eval scores for every revision.
- Canary and shadow evaluation run a new prompt against live traffic before full rollout.