A human never says how good — only which is better. This machine turns a pile of "A is better than B" into a single number per item. Under the Bradley–Terry model, P(A≫B)=σ(rA−rB); fitting the rewards r is gradient ascent on a concave log-likelihood, so it provably climbs to its target — not stochastic guessing. Preferences go in, the rewards come out. The blue team builds and defends it; the red team tries to break it.
source Christiano, Leike, Brown, Martic, Legg, Amodei, Deep Reinforcement Learning from Human Preferences (NeurIPS 2017) — arxiv.org/abs/1706.03741. Rendered, not quoted.
Give every item i a hidden scalar reward ri. The chance a human prefers a to b depends only on the difference:
Fit r by maximizing the log-likelihood of the observed comparisons. That objective is concave in r — gradient ascent has no bad local maxima, so it reaches the fit. Live rewards for the current step:
| item | reward r | magnitude |
|---|
Maximizing this log-likelihood is minimizing the cross-entropy between the human's hard label (a beat b, so target = 1) and the model's P(a≫b). One preference pair = one binary cross-entropy term.
This scalar r is the human-rater stand-in that RLHF then optimizes against, and the same object the-dpo folds directly into a policy loss. Each sphere is the next one's premise.
The blue team's live check: re-fit the model from scratch and confirm the four invariants — σ-symmetry, monotone ascent, shift-invariance, and that the fit orders every preference. If red flips ascent to descent, this badge is where it shows.
The only input is a set of ordinal judgments — no scores, no magnitudes. A human was shown two items and said which is better. Here, four candidate responses A B C D and five pairwise verdicts:
| # | human judged | meaning |
|---|
That is all the machine sees: which ≫ which. It never receives a number. The reward scale below is invented to explain these choices — and is pinned down only up to an additive constant.
Gradient ascent (η=0.25) on the concave log-likelihood. Every step the log-likelihood rises toward its ceiling of 0; nothing is looked up.
Step the ascent, change the init, or query any pair — every number is computed live from σ and the gradient, never stored.
What the machine produces, proven: a single scalar reward per item that orders every one of the given preferences, and predicted preference probabilities P(a≫b)=σ(ra−rb) for any pair. The rewards are identified only up to a constant — add the same number to all four and nothing observable changes.
The blue team's witness (left) re-fits and confirms these live; the red team (right) tries to make the ordering wrong.
A single scalar also forces a total order on values that may be genuinely incomparable or cyclic. If human labels are intransitive (A≫B, B≫C, C≫A) there is no Bradley–Terry fit at all — the likelihood saturates below perfect and the "true reward" it recovers is a fiction.
"The reward model learns the true reward." Cut. It learns rewards up to an additive constant, consistent only with the comparisons seen. Absolute reward is not identifiable — only differences are.
"Reward is an absolute quantity." Cut. Shift every r by the same c and every P(a≫b) is unchanged (window 8, verified to 1e-12). The zero point is a gauge choice, here fixed by re-centering to mean 0.
"More preference data always helps." Kept, corrected. Consistent data sharpens the fit; inconsistent (intransitive) data has no perfect optimum — the concave climb still converges, but to a compromise, not the truth.
The red team's move: flip the update from ascent to descent — minimize the log-likelihood instead of maximizing it. The learned ordering inverts and starts violating the very preferences it was fit on. The blue team's witness (window 7) is watching.
Descend on the objective and the rewards run the wrong way — the fit stops ordering the preferences, the witness recomputes, disagrees, and turns red. Nothing is faked; the attack is real and it is caught.