Evaluation is a control loop, not a report card
A benchmark score is only useful if it drives a decision. The evaluation loop makes that explicit: state the claim, build an instrument that can falsify it, measure, diagnose the failure, change exactly one thing, and re-measure under the identical protocol.
Attach every score to a decision
Ship, fix, or investigate. If no action follows from a metric, you have built a dashboard — not an evaluation.
Change one variable
Model, prompt, decoding, and data all move the score. Change one at a time or the result is unattributable.
Freeze the protocol
Prompt format, few-shot count, sampling temperature, and the item set are part of the measurement. Changing any of them invalidates the comparison.
An observed improvement mixes the true change with protocol drift and sampling noise. A 1-point gain on 200 items has a 95% half-width of ±6.3 points — larger than the effect it claims to show.
- Eval-driven teams keep a small 'north star' suite stable for months so year-over-year comparisons remain valid.
- Releases are blocked on regressions in the north star, while exploratory suites iterate freely.