EVALUATION

MODULE 6: HUMAN & ARENA EVALUATION · SCENE 16

Annotation Protocols & Guidelines

Instructions, gold items, adjudication, and rater calibration decide whether human labels are a measurement or an opinion pool.

GOLD-ITEM GATEKEEP ≥ 0.803/6 RATERS · 50%
rater-010.98 ✓rater-020.92 ✓rater-030.85 ✓rater-040.72 ✗rater-050.61 ✗rater-060.55 ✗GOLD LINE = QUALIFICATION · FAILING RATERS PRODUCE OPINIONS, NOT LABELS

Raise the gate and the pool shrinks but the labels harden. Adjudicate what remains instead of averaging it away.

ARCHITECTURE & FORMULATIONModule 6: Human & Arena Evaluation

Human labels are an instrument, not an opinion pool

Human evaluation quality is determined before any labeling starts — by the guideline, the task framing, the rater training, and the quality-control stream. A vague guideline produces labels that cannot be aggregated no matter how many raters you add.

Guidelines are the spec

Every decision the rater must make should be documented with examples, including edge cases and tie handling.

Gold items

Seed known-answer items into the stream to detect raters who are not reading or not following the guide.

Adjudication

Resolve disagreements through a defined process rather than majority-voting them into noise.

MATHEMATICAL FORMULATION · RATER RELIABILITY GATE
keep_rater if gold_accuracy ≥ threshold

A rater answering seeded gold items with 60% accuracy is not producing usable labels regardless of how many items they label.

REAL-WORLD PRODUCTION ENGINEERING
  • Annotation platforms run qualification rounds and embed gold items continuously, not just at onboarding.
  • Safety data labeling uses detailed policy rubrics with escalation paths for ambiguous cases.
← 15 · Judge Bias: Position, Length, Self
17 · Inter-Annotator Agreement →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]