EVALUATION

MODULE 4: AUTOMATIC SCORING · SCENE 10

Exact Match & Token F1

Normalization first, then exact match or token-level F1 — cheap, deterministic, and blind to meaning it was not told to check.

PREDICTIONGOLD · ParisEXACT ✓ 1TOKEN F1 · 1.000
NORMALIZED STRINGS IDENTICAL — FULL CREDITTOKEN F1 = 1.000 — PARTIAL CREDIT FOR SHARED TOKENS (“paris.” vs “Paris”)

Try “The Paris” or “paris france”: normalization absorbs the first, F1 forgives the second. The scorer is part of the item.

ARCHITECTURE & FORMULATIONModule 4: Automatic Scoring

Normalize first, then exact match or token F1

String scoring is the cheapest, most deterministic scorer available — and the most sensitive to formatting. Normalization (case, whitespace, punctuation, articles) is applied before comparison, because 'The Eiffel Tower.' and 'eiffel tower' should not be different answers. Token F1 extends the idea with partial credit over bags of words.

Normalization is the scorer

Every normalization choice changes the score. Fix it in the protocol and keep it constant across models.

Exact match

All-or-nothing after normalization. Correct for multiple choice, named entities, and numeric answers.

Token F1

Multiset overlap between prediction and reference tokens. Tolerates word-order changes and gives partial credit.

MATHEMATICAL FORMULATION · TOKEN F1
F1 = 2 · precision · recall / (precision + recall)

Prediction 'the quick brown fox' against 'the quick fox': precision 3/4, recall 3/3, F1 ≈ 0.857. The extra adjective costs precision without costing recall.

REAL-WORLD PRODUCTION ENGINEERING
  • SQuAD reporting uses normalized exact match plus token F1 as the two standard extractive-QA metrics.
  • Numeric answers are compared after rounding and unit normalization, never as raw strings.
← 09 · Private & Rotating Sets
11 · BLEU/ROUGE and Their Limits →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]