RLHF: PPO with a KL tether
With a trained reward model, PPO optimizes the policy to maximize reward — but an unconstrained optimizer will exploit every flaw in the reward model. The KL penalty against the frozen reference policy is what keeps alignment optimization honest.
Four Models in Memory
Policy (training), reference (frozen), reward model (frozen), value/critic (training). That quadruple memory bill is why PPO is expensive.
Clipped Objective
PPO clips the probability ratio to [1−ε, 1+ε] so one batch cannot move the policy far enough to escape the trust region.
GAE
Generalized Advantage Estimation blends multi-step TD residuals with parameter λ, trading bias against variance in the credit assignment.
The clip keeps updates local; the KL term pays a per-token rent for drifting from the reference policy. β is the dial between alignment gains and reward-model exploitation.
- InstructGPT used β around 0.01–0.1; too small and responses drift into reward-hacked degeneracy within a few hundred steps.
- PPO at scale typically runs the reward model and reference policy on separate GPUs and batches rollouts — the pipeline is a mini distributed-systems project.