Reward Models

Lesson 2 of 4 in RLHF: Learning from Preferences.

A reward model is a judge built from a language model. Take a copy of the SFT model (or another model of similar pedigree), remove the next-token output head, and bolt on a head that emits one number: a scalar score for a (prompt, response) pair. High score means “a human would probably prefer this”; low score means they probably would not. That is the entire interface — text in, one number out.

Training it is where the preference data earns its keep. You cannot ask labelers for absolute scores — “rate this answer 7.2 out of 10” drifts wildly between people and even within one person across a session. Pairwise judgments are the stable currency. So the reward model trains on pairs: for each prompt, a chosen response and a rejected one, and a Loss that pushes the chosen response’s score above the rejected one’s. When labelers rank K responses at once, every ranking unrolls into K·(K−1)/2 such pairs.

One training step for a reward model

  1. Prompt from the collection set

    Drawn from a distribution meant to resemble real usage — in practice, whatever the lab sampled. Remember this when the model later meets your prompts.

  2. Model generates candidate responses

    Two or more responses to the same prompt, produced by Sampling from the SFT model (or several models).

  3. Human picks the better response

    A comparative judgment, guided by labeling instructions — the written-down values the annotators are told to apply.

  4. Reward model scores both responses

    Same weights score chosen and rejected independently; each gets a scalar.

  5. Loss: chosen must outscore rejected

    The gap between the two scores is pushed apart — the Bradley–Terry-style objective in the deep dive below.

  6. Update reward-model weights

    Repeat over the whole preference dataset. The result: a frozen, queryable stand-in for the labelers.

Once trained, the reward model is a remarkable artifact: human judgment, amortized. Collecting a comparison costs an annotator’s attention once; querying the reward model costs a forward pass, millions of times, on responses no human will ever see. That is what makes stage three affordable at all.

But be precise about what it learned. Not “what is good.” Not “what your users want.” It learned which of two responses these particular labelers, following these particular instructions, on this particular prompt distribution, tended to prefer — compressed into a single scalar. Every word of that sentence is a place where the proxy can diverge from what you actually care about: labelers disagree with each other and with your users, instructions encode a vendor’s policy choices, and the prompt distribution never quite matches production traffic.

The Bradley–Terry objective on preference pairs

The training objective comes from the Bradley–Terry model, a classic statistical model of paired comparisons: assume each item has a latent quality score, and the probability that one beats the other grows with the score difference, squashed through a sigmoid. Applied here, with r(x, y) the reward model’s score for response y to prompt x:

P(y_w ≻ y_l | x) = σ( r(x, y_w) − r(x, y_l) )

where y_w is the chosen (winning) response, y_l the rejected one, and σ the logistic sigmoid (the two-option cousin of the Softmax). Training maximizes the likelihood of the observed human choices — equivalently, minimizes:

L = −E[ log σ( r(x, y_w) − r(x, y_l) ) ]

Three properties worth internalizing:

  • Only differences matter. Add any constant to every score and nothing changes — the model learns a relative scale, not an absolute one. A reward of 3.7 means nothing by itself; it only means something compared to another response to the same prompt.
  • The gap encodes confidence. A large score difference puts the sigmoid near 1: the model is claiming humans would almost always pick the winner. Noisy, split human votes push scores together.
  • Rankings unroll into pairs. A labeler ranking K responses produces K·(K−1)/2 training pairs from one screenful of attention — part of why comparisons are such an efficient way to spend annotation budget (this is the setup used at scale in Ouyang et al. 2022, arXiv:2203.02155, following Stiennon et al. 2020, arXiv:2009.01325).

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.