Percent agreement is inflated by chance; Cohen's kappa subtracts it. If raters cannot agree, the task itself may be ill-defined.
RATER A POS RATE0.60RATER B POS RATE0.55OBSERVED AGREE0.80
Skewed tasks inflate raw agreement: two raters who mostly say “yes” will match by luck. Kappa subtracts the luck.
ARCHITECTURE & FORMULATIONModule 6: Human & Arena Evaluation
Agreement must be corrected for chance
Two raters will agree by chance on any balanced binary task roughly half the time. Percent agreement therefore overstates reliability. Cohen's kappa subtracts the chance component, exposing whether raters are actually using the same construct. Low kappa means the task or the guideline is ill-defined.
Percent agreement
The raw fraction of items where raters match. Simple, intuitive, inflated by chance and by skewed label distributions.
Cohen's kappa
κ = (p_o − p_e) / (1 − p_e), where p_e is the agreement expected if labels were independent.
Interpretation
κ < 0.4 signals a problem; 0.6–0.8 is substantial; above 0.8 is strong. Always report it with the labels.
MATHEMATICAL FORMULATION · COHEN'S KAPPA
κ = (p_o − p_e) / (1 − p_e)
Observed agreement 0.8 against chance agreement 0.5 gives κ = 0.6. The same 0.8 with a harder task (chance 0.2) gives κ = 0.75 — reliability depends on the task's base rates.
REAL-WORLD PRODUCTION ENGINEERING
Krippendorff's alpha generalizes kappa to more than two raters and missing data.
A low kappa is usually fixed by tightening the guideline, not by adding raters.