Agreement must be corrected for chance
Two raters will agree by chance on any balanced binary task roughly half the time. Percent agreement therefore overstates reliability. Cohen's kappa subtracts the chance component, exposing whether raters are actually using the same construct. Low kappa means the task or the guideline is ill-defined.
Percent agreement
The raw fraction of items where raters match. Simple, intuitive, inflated by chance and by skewed label distributions.
Cohen's kappa
κ = (p_o − p_e) / (1 − p_e), where p_e is the agreement expected if labels were independent.
Interpretation
κ < 0.4 signals a problem; 0.6–0.8 is substantial; above 0.8 is strong. Always report it with the labels.
Observed agreement 0.8 against chance agreement 0.5 gives κ = 0.6. The same 0.8 with a harder task (chance 0.2) gives κ = 0.75 — reliability depends on the task's base rates.
- Krippendorff's alpha generalizes kappa to more than two raters and missing data.
- A low kappa is usually fixed by tightening the guideline, not by adding raters.