AI ENGVisual Encyclopedia

MODULE 8: REGRESSION GATES & MONITORING · SCENE 24

Online Monitoring & Drift Detection

Score distributions move as inputs and populations shift. Sample production, track drift, and gate deploys on live evidence.

BASELINE MEAN0.70BASELINE σ0.05CURRENT MEAN0.63DRIFT · 1.40σ
BASELINE 0.70LIVE 0.63WATCH — MOVEMENT WORTH A CANARY CHECK BEFORE FULL RELEASE

No code changed and quality still moved — because the inputs did. Sample production continuously, or the offline eval becomes history.

TECHNICAL BREAKDOWNModule 8: Regression Gates & Monitoring

Production inputs move; offline scores age

Offline evals are a proxy for production, and the proxy degrades as the input population shifts. Online monitoring samples live traffic, scores it against expectations (automatically or with an LLM judge), and tracks distribution drift. A standardized drift score turns 'it feels worse' into an alert.

Sample and score

Continuously sample production requests and score a subset to estimate live quality.

Distribution drift

Track the mean and spread of inputs and scores. A shift in either signals that the offline eval no longer represents traffic.

Bound the blast radius

Canary deploys expose a small fraction of traffic first, so drift and failures surface before full release.

MATHEMATICAL FORMULATION · STANDARDIZED DRIFT
drift = |current_mean − baseline_mean| / baseline_std

A drift of 1.4 standard deviations is a large population shift — the kind that can move quality without any code change, because the inputs changed.

REAL-WORLD PRODUCTION ENGINEERING
  • Production monitors combine task-success proxies, judge scores, and input-distribution alerts.
  • Shadow deployments score candidate models on live prompts before switching any user traffic.
← 23 · Prompt & Model Regression Testing
ATLAS →