No reward model: the policy's log-ratio vs the reference IS the reward. IPO adds a margin target; KTO uses win/loss labels; SimPO drops the reference and length-normalizes.
DPO's insight is algebraic: the closed-form solution of the KL-constrained reward maximization lets you rewrite the reward in terms of the policy itself. Substitute that into Bradley-Terry and the reward model disappears — you optimize the policy directly on preference pairs.
Implicit Reward
r(x, y) = β·log(π(y|x)/π_ref(y|x)) + constant. The policy's log-ratio against the reference IS the reward.
Simpler Pipeline
Two models in memory (policy + reference), one supervised loss, no rollouts. DPO is to PPO what a sedan is to a crane.
The Heirs
IPO adds a margin target to fight overfitting; KTO drops pairs for win/loss labels; SimPO drops the reference model and length-normalizes instead.
With β=0.1, pushing the chosen response's implicit reward 5 nats above the rejected one gives σ(0.5) ≈ 0.62 → loss 0.48; separation must reach 20+ nats before the loss approaches zero.
REAL-WORLD PRODUCTION ENGINEERING
DPO's biggest practical win is operational: one stage, offline data, standard SFT infrastructure.
Known failure mode: unbounded log-ratios push chosen and rejected likelihoods both DOWN — monitor absolute log-probs, not just the margin. SimPO/IPO exist partly for this.