Skipping the Reward Model

Lesson 1 of 3 in DPO and the Direct Methods.

The RLHF module ended in machinery. To learn from preference data, the classic pipeline first trains a separate reward model on the comparisons, then runs an online reinforcement-learning loop: sample fresh responses from the policy, score them with the reward model, update with PPO while a KL penalty tethers the policy to a frozen reference. Four networks in play at once, live text generation inside the training loop, and a family of RL hyperparameters that can each destabilize the run.

Direct Preference Optimization (Rafailov et al. 2023, arXiv:2305.18290) asks the question that titles its paper: what if your language model is secretly a reward model? The RLHF objective — maximize reward while staying close to the reference — turns out to have a known closed-form solution, and that solution can be run backwards: any policy, paired with its reference model, implicitly defines a reward function. If the policy already encodes a reward, you do not need to build one. DPO takes the exact same preference pairs — prompt, chosen response, rejected response — and trains the policy on them directly, with a simple classification-style Loss. Same data. Same theoretical target. Two fewer subsystems.

Two routes from preference data to a tuned policy

  1. Preference dataset

    Records of the form (prompt x, chosen response y⁺, rejected response y⁻) — typically labeled by humans or, in RLAIF, by a model. Both routes start here.

  2. Train a reward model

    A separate network learns to score responses so that chosen beats rejected — the subject of the previous module. It must be trained, evaluated, and kept running for the whole RL phase.

  3. Sample fresh responses from the policy

    Online generation inside the training loop — thousands of rollouts. The expensive, operationally fragile inner loop.

  4. Score samples with the reward model

    Every sampled response gets a scalar reward. If the reward model has blind spots, the policy will find them — reward hacking.

  5. PPO update with KL penalty

    A reinforcement-learning step nudges the policy toward higher reward while the KL term keeps it near the frozen reference model.

  6. Compute the DPO loss on the pairs

    A classification-style loss compares the policy’s log-probabilities on chosen vs rejected responses against a frozen reference model. No sampling. No reward model.

  7. Standard gradient step

    The loop is shaped like supervised fine-tuning: batches in, gradients out, on ordinary training infrastructure.

  8. Preference-tuned policy

    Both routes aim at the same theoretical target: the best policy within a KL leash of the reference.

Look at what fell out of the right-hand route. The reward model is gone — no second network to train, no separate evaluation to run, nothing extra to host in GPU memory during training. The online sampling loop is gone — no generation inside training, which was both the dominant cost and the dominant source of instability. The RL machinery is gone — no value function, no rollout batching, no PPO-specific hyperparameters to tune.

What that buys is easy to state: a preference-tuning run now has the shape of supervised fine-tuning. Batches of pairs go in, gradients come out, and the whole thing runs on the same infrastructure and skill set a team already uses for SFT. Runs are more stable — there is no learned reward whose flaws the policy can chase mid-run, and no rollout variance — and dramatically cheaper in both compute and engineering attention. This is not a free lunch (lesson three is the bill), but it is a real and large simplification, and it is why direct methods spread through the open-model world within months of publication.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.