AI ENGVisual Encyclopedia

MODULE 4: AUTOMATIC SCORING · SCENE 12

Execution & Programmatic Grading

Unit tests, regexes, and checkers turn open-ended output into a pass/fail oracle — the backbone of code and math eval.

SAMPLES n10CORRECT c5k5
EXECUTED, NOT READ — 5/10 SAMPLES PASS THE TESTSpass@1 (RELIABILITY) · 0.500pass@5 (REACHABLE CAPABILITY) · 0.9960.50 RELIABLE, 1.00 REACHABLE — THE GAP IS WHAT SAMPLING BUYS

A verifier cannot be charmed by fluent prose. But extract the code block wrong and the failure is yours, not the model's.

TECHNICAL BREAKDOWNModule 4: Automatic Scoring

Execute the answer: verifiers as ground truth

For code and math, the answer can be checked by running it. Programmatic grading extracts a candidate program or expression, executes it against tests or a reference implementation, and returns a deterministic pass/fail. This is the strongest form of automatic scoring because it cannot be fooled by fluent prose.

Robust extraction

Find the code block or final answer before grading. Extraction failures masquerade as model failures and are a common source of bogus scores.

Test harnesses

Unit tests, property checks, and reference implementations provide binary or partial credit.

pass@k

When sampling k solutions, report the fraction of problems solved by at least one sample.

MATHEMATICAL FORMULATION · PASS@K
pass@k = 1 − C(n − c, k) / C(n, k)

With 10 samples and 5 correct, pass@1 = 0.5 while pass@5 ≈ 0.996. pass@k measures reachable capability; pass@1 measures reliability.

REAL-WORLD PRODUCTION ENGINEERING
  • HumanEval and MBPP execute generated Python against hidden tests and report pass@1 and pass@k.
  • SWE-bench verifies patches by running a repository's own test suite in an isolated container.
← 11 · BLEU/ROUGE and Their Limits
13 · Rubric Design & Pointwise Scoring →