RLHF, without the RL. No separate reward model, no reward loop — the reward is read off the policy itself as a log-ratio r(y)=β·log(π(y)/π_ref(y)), and a single classification loss L=−log σ(r(yw)−r(yl)) pushes the preferred completion above the dispreferred one. Down the center, data flows: the completions and π_ref go in, the engine computes the implicit rewards and the loss, the proven result comes out. The blue team builds and defends it; the red team tries to break it.
source Rafailov, Sharma, Mitchell, Ermon, Manning, Finn, Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023) — arxiv.org/abs/2305.18290. Rendered, not quoted.
There is no reward network. The implicit reward is read off the policy, relative to a frozen reference:
r(y) = β · log( π(y) / π_ref(y) )
A preferred yw beats a dispreferred yl with probability σ(r(yw)−r(yl)), and the whole objective is one classifier: L = −log σ( r(yw) − r(yl) ). Live implicit rewards for the current π:
| y | π_ref | π | r(y)=β·log(π/π_ref) |
|---|
Its neighbour the-reward-model fits an explicit Bradley–Terry reward r*(y) to the same preferences, then RL-optimises the KL-constrained objective whose optimum is π*(y) ∝ π_ref(y)·exp(r*(y)/β).
Invert that closed form: r*(y) = β·log(π*(y)/π_ref(y)) + β·log Z. The reward is the log-ratio. So the separate model is unnecessary — it can be folded into the policy and trained by one classification loss. Same objective, no RL loop. The witness proves the two rankings coincide.
The blue team's live check: at π=π_ref the win-prob is exactly 0.5; the sigmoid argument equals r(yw)−r(yl); the implicit-reward ranking matches Bradley–Terry; and stepping π toward yw lowers L. If red tampers, this badge is where it shows.
One prompt — "Summarize the meeting" — and three candidate completions, with a frozen reference policy π_ref and a fixed temperature β = 0.5:
| y | completion | π_ref |
|---|---|---|
| y_a | faithful one-line summary | 0.50 |
| y_b | generic filler | 0.30 |
| y_c | invented details | 0.20 |
A preference pair is yw (preferred) over yl (dispreferred). Default: y_a ≻ y_c. This — plus the sliders on π below — is what you feed the panel.
Sliders are normalised to a probability distribution. "DPO-optimal" sets π to the closed-form π* for the Bradley–Terry reward [+1,0,−1].
Every reward and the loss are computed live from r(y)=β·log(π/π_ref) — never looked up.
What the machine produces, proven: the implicit rewards, the win-probability σ(margin), and the scalar loss L — the current values shown live below. And the equivalence the paper names: DPO's implicit reward equals the Bradley–Terry reward up to an additive constant, so it induces the same ranking and the same preference probabilities — with no reward model trained.
The blue team's witness (left) proves these live; the red team (right) tries to make the loss reward the wrong completion.
There is also no online exploration: with a fixed offline dataset DPO can only re-weight completions the reference already puts mass on. It is not "RLHF made safe" — it is RLHF's objective made convex in the reward, which is a different, narrower guarantee.
"DPO has no reward model." Corrected. It has no separately trained one — the reward is still there, defined implicitly as β·log(π/π_ref). "Your language model is secretly a reward model."
"DPO always raises the chosen response's probability." Cut. Only the gap is guaranteed to grow; π(yw) itself can decrease. A documented failure mode.
"DPO removes the KL constraint." Cut. β is exactly the KL temperature; it is baked into the reward's scale, not deleted. Set β→0 and the reward vanishes.
The red team's move: swap yw and yl inside the loss — train toward the dispreferred completion. The loss now rewards the wrong answer, and the blue team's witness (window 7) is watching.
Swap the pair and "raising π(y_w) lowers L" flips — moving mass toward the preferred completion now increases the loss. The witness recomputes, disagrees, and turns red. Nothing is faked; the attack is real and it is caught.