EVALUATION

MODULE 8: REGRESSION GATES & MONITORING · SCENE 22

Eval-Driven CI Gates

Per-capability regression budgets turn evaluation into a release gate: no ship when a tested axis falls past its allowance.

MATH AFTER0.830CODE AFTER0.750SAFETY AFTER0.660ALLOWANCE±0.020
MATH 0.82 → 0.83 · Δ +0.010✓ WITHIN BUDGETCODE 0.74 → 0.75 · Δ +0.010✓ WITHIN BUDGETSAFETY 0.69 → 0.66 · Δ -0.030✗ BREACH — BLOCK⛔ RELEASE BLOCKED — ONE BROKEN AXIS VETOES A RISING AVERAGE

Nudge SAFETY below its allowance and the average can still climb. A global threshold ships the breakage; a per-axis gate does not.

ARCHITECTURE & FORMULATIONModule 8: Regression Gates & Monitoring

Turn evaluation into a release gate

An eval that nobody blocks on is documentation. A regression gate attaches a per-capability allowance to the tested axes and fails the build when any axis falls past it. Per-capability budgets matter because a rising average can hide a broken axis — the exact failure a global threshold misses.

Per-capability allowances

Math, code, safety, and instruction-following each get their own tolerance, set from historical noise.

Fail on the axis

Even if the composite improves, a single axis past its allowance blocks the release.

Review the thresholds

Gate thresholds are code: versioned, reviewed, and changed deliberately.

MATHEMATICAL FORMULATION · GATE BREACH
breach if (after − before) < −allowance

A capability allowed to move ±2 points fails when it drops 3. Setting the allowance too tight makes the gate flaky; too loose and real regressions ship.

REAL-WORLD PRODUCTION ENGINEERING
  • CI pipelines run a fixed eval subset on every prompt or model change and block the merge on regression.
  • Gate allowances are usually derived from the measured run-to-run variance of each suite.
← 21 · Bootstrap & Multiple Comparisons
23 · Prompt & Model Regression Testing →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]