Ask a human 'is A better than B?' and the answer is noisy. Bradley-Terry turns pairwise comparisons into a scalar reward per response, where the probability A wins is a sigmoid of the reward gap. It is the statistical foundation under RLHF, DPO, and every variant.
The Model
Each response gets a latent reward r. P(y_w ≻ y_l) = σ(r_w − r_l). Fit rewards by maximizing the likelihood of observed human choices.
The Loss
Reward-model training minimizes −log σ(r_w − r_l) over pairs. A well-fit model ranks held-out pairs correctly; its accuracy is the ceiling of everything downstream.
Why Pairs
Pairwise 'which is better' is far more reliable than absolute 1–10 scoring. Humans are consistent comparators, inconsistent scorers.
MATHEMATICAL FORMULATION · PREFERENCE LOSS
L_RM = −log σ(r_w − r_l) = −log P(y_w ≻ y_l)
Reward margin of 2.2 gives σ(2.2) ≈ 0.90 → loss 0.105. Margin 0 gives loss log 2 ≈ 0.693, the maximum a binary comparison can contribute.
REAL-WORLD PRODUCTION ENGINEERING
Reward-model accuracy on held-out pairs is typically 65–75% — every downstream RLHF run inherits that noise ceiling.
Annotator disagreement is data: many teams train an ensemble of reward models and treat disagreement as an uncertainty signal.