EVALUATION

MODULE 1: EVAL FOUNDATIONS · SCENE 01

The Evaluation Loop

Measure → diagnose → change → re-measure. Evaluation is not a final report; it is the control loop that tells you whether the system moved.

LOOP STAGE · READY
1 · MEASURE○ WAITING→2 · DIAGNOSE○ WAITING→3 · CHANGE ONE THING○ WAITING→4 · RE-MEASURE○ WAITINGONE LOOP = ONE ATTRIBUTABLE DECISION · PROTOCOL FROZEN THROUGHOUT

An unevaluated model is a rumor. RUN to walk the control loop.

ARCHITECTURE & FORMULATIONModule 1: Eval Foundations

Evaluation is a control loop, not a report card

A benchmark score is only useful if it drives a decision. The evaluation loop makes that explicit: state the claim, build an instrument that can falsify it, measure, diagnose the failure, change exactly one thing, and re-measure under the identical protocol.

Attach every score to a decision

Ship, fix, or investigate. If no action follows from a metric, you have built a dashboard — not an evaluation.

Change one variable

Model, prompt, decoding, and data all move the score. Change one at a time or the result is unattributable.

Freeze the protocol

Prompt format, few-shot count, sampling temperature, and the item set are part of the measurement. Changing any of them invalidates the comparison.

MATHEMATICAL FORMULATION · SIGNAL VS NOISE
Δ_observed = Δ_true + ε_protocol + ε_sampling

An observed improvement mixes the true change with protocol drift and sampling noise. A 1-point gain on 200 items has a 95% half-width of ±6.3 points — larger than the effect it claims to show.

REAL-WORLD PRODUCTION ENGINEERING
  • Eval-driven teams keep a small 'north star' suite stable for months so year-over-year comparisons remain valid.
  • Releases are blocked on regressions in the north star, while exploratory suites iterate freely.
← WORLD HOME
02 · Capability vs Alignment Evals →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]