Bradley-Terry: preferences as probabilities
Ask a human 'is A better than B?' and the answer is noisy. Bradley-Terry turns pairwise comparisons into a scalar reward per response, where the probability A wins is a sigmoid of the reward gap. It is the statistical foundation under RLHF, DPO, and every variant.
The Model
Each response gets a latent reward r. P(y_w ≻ y_l) = σ(r_w − r_l). Fit rewards by maximizing the likelihood of observed human choices.
The Loss
Reward-model training minimizes −log σ(r_w − r_l) over pairs. A well-fit model ranks held-out pairs correctly; its accuracy is the ceiling of everything downstream.
Why Pairs
Pairwise 'which is better' is far more reliable than absolute 1–10 scoring. Humans are consistent comparators, inconsistent scorers.
Reward margin of 2.2 gives σ(2.2) ≈ 0.90 → loss 0.105. Margin 0 gives loss log 2 ≈ 0.693, the maximum a binary comparison can contribute.
- Reward-model accuracy on held-out pairs is typically 65–75% — every downstream RLHF run inherits that noise ceiling.
- Annotator disagreement is data: many teams train an ensemble of reward models and treat disagreement as an uncertainty signal.