AI ENGVisual Encyclopedia

MODULE 4: AUTOMATIC SCORING · SCENE 10

Exact Match & Token F1

Normalization first, then exact match or token-level F1 — cheap, deterministic, and blind to meaning it was not told to check.

PREDICTIONGOLD · ParisEXACT ✓ 1TOKEN F1 · 1.000
NORMALIZED STRINGS IDENTICAL — FULL CREDITTOKEN F1 = 1.000 — PARTIAL CREDIT FOR SHARED TOKENS (“paris.” vs “Paris”)

Try “The Paris” or “paris france”: normalization absorbs the first, F1 forgives the second. The scorer is part of the item.

TECHNICAL BREAKDOWNModule 4: Automatic Scoring

Normalize first, then exact match or token F1

String scoring is the cheapest, most deterministic scorer available — and the most sensitive to formatting. Normalization (case, whitespace, punctuation, articles) is applied before comparison, because 'The Eiffel Tower.' and 'eiffel tower' should not be different answers. Token F1 extends the idea with partial credit over bags of words.

Normalization is the scorer

Every normalization choice changes the score. Fix it in the protocol and keep it constant across models.

Exact match

All-or-nothing after normalization. Correct for multiple choice, named entities, and numeric answers.

Token F1

Multiset overlap between prediction and reference tokens. Tolerates word-order changes and gives partial credit.

MATHEMATICAL FORMULATION · TOKEN F1
F1 = 2 · precision · recall / (precision + recall)

Prediction 'the quick brown fox' against 'the quick fox': precision 3/4, recall 3/3, F1 ≈ 0.857. The extra adjective costs precision without costing recall.

REAL-WORLD PRODUCTION ENGINEERING
  • SQuAD reporting uses normalized exact match plus token F1 as the two standard extractive-QA metrics.
  • Numeric answers are compared after rounding and unit normalization, never as raw strings.
← 09 · Private & Rotating Sets
11 · BLEU/ROUGE and Their Limits →