EVALUATION

MODULE 8: REGRESSION GATES & MONITORING · SCENE 23

Prompt & Model Regression Testing

A frozen golden set catches the quiet breakage a prompt tweak or fine-tune causes before users do.

CLICK ITEMS TO FLIP PASS ⇄ FAILADVERSE RATE · 16.7% (2/12)FLIPPED · #3, #8
#1 ✓ PASS#2 ✓ PASS#3 ✗ FAIL#4 ✓ PASS#5 ✓ PASS#6 ✓ PASS#7 ✓ PASS#8 ✗ FAIL#9 ✓ PASS#10 ✓ PASS#11 ✓ PASS#12 ✓ PASS2 QUIET BREAKS THE AGGREGATE HIDES — DIFF ITEM BY ITEM

Two items improving can mask two breaking. A prompt edit that “only rewords” is a code change with no compiler — re-run the goldens.

ARCHITECTURE & FORMULATIONModule 8: Regression Gates & Monitoring

A frozen golden set catches quiet breakage

Prompt changes are code changes with no compiler. Editing one instruction to fix an edge case can silently break fifty other cases. A golden set — a frozen, versioned collection of representative inputs and expected behaviors — is re-run on every change and diffed item by item.

Freeze and version

The golden set is immutable per version. New cases are added in a new version so old comparisons stay valid.

Diff, don't average

Aggregate scores hide swaps: two items improving can mask two breaking. Inspect per-item flips.

Tie prompt to evidence

Every prompt version records the eval run that approved it, so regressions can be bisected.

MATHEMATICAL FORMULATION · ADVERSE REGRESSION RATE
rate = regressed_items / tested_items

If 40 of 2,000 golden items flip from pass to fail, the adverse regression rate is 2% — small in aggregate, but decisive if those items cover a critical capability.

REAL-WORLD PRODUCTION ENGINEERING
  • Prompt-management systems store prompt versions alongside eval scores for every revision.
  • Canary and shadow evaluation run a new prompt against live traffic before full rollout.
← 22 · Eval-Driven CI Gates
24 · Online Monitoring & Drift Detection →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]