The DPO Objective
Lesson 2 of 3 in DPO and the Direct Methods.
The DPO loss reads like a sentence once you know its parts. For each training example — a prompt, a chosen response, a rejected response — it asks one question: does the policy favor the chosen response over the rejected one more strongly than the reference model does? If yes, the example contributes almost nothing. If no, it contributes a large loss and a large gradient. Everything in the formula is bookkeeping around that question.
Two dials shape the answer. The frozen reference model — usually the SFT checkpoint the run started from — defines the baseline: favoritism is measured as change from the reference, not in absolute terms, which is how the KL leash survives the reward model’s removal. And β sets the exchange rate: how expensive it is for the policy to drift away from that reference in pursuit of the preferences.
Notice what is not in the loss: nothing is sampled, and no separate network scores anything. The loss is computed entirely from log-probabilities that the policy and the reference assign to text that already exists in the dataset — which is why a DPO step costs about as much as a supervised fine-tuning step.
Bar chart with four bars showing illustrative probabilities a model assigns to two candidate responses. Under the reference model, the chosen response has probability 0.22 and the rejected response 0.18 — nearly equal. After DPO training, the policy assigns 0.38 to the chosen response and 0.05 to the rejected one, showing probability mass moving toward the chosen response relative to the reference.
Now name the pieces. For each response, DPO computes a log-ratio: the log-probability the policy assigns to the response, minus the log-probability the reference assigns to it. Read it as “how much has training moved the policy on this response?” — positive means the policy now likes it more than the reference did, negative means less. That log-ratio, scaled by β, is DPO’s implicit reward.
The loss then takes the implicit reward of the chosen response minus that of the rejected one — a margin — and feeds the margin through a log-sigmoid. That is exactly the loss of logistic regression on the question “which response won?”. So DPO is, literally, training a classifier of human preferences whose score function is built out of the policy’s own probabilities. The policy is the reward model — that is the secret in the paper’s title, now in equation form.
The DPO loss, term by term
The objective, as published (Rafailov et al. 2023):
L_DPO(π_θ; π_ref) = −E_(x, y⁺, y⁻) [ log σ( β·( log(π_θ(y⁺|x) / π_ref(y⁺|x)) − log(π_θ(y⁻|x) / π_ref(y⁻|x)) ) ) ]
The atoms. π_θ(y|x) is the probability the trainable policy assigns to the whole response y given prompt x — in practice a sum of per-token log-probabilities from one forward pass. π_ref is the same quantity under the frozen reference model. σ is the sigmoid; y⁺ and y⁻ are the chosen and rejected responses.
The log-ratios. log(π_θ(y|x)/π_ref(y|x)) measures the policy’s movement on one response relative to the reference. Multiplied by β, it is the implicit reward r̂(x,y) = β·log(π_θ(y|x)/π_ref(y|x)) — up to a prompt-only constant that is about to cancel.
The margin and the sigmoid. The difference r̂(x,y⁺) − r̂(x,y⁻) is a reward margin; −log σ(margin) is the binary logistic loss saying the chosen response should win. Large positive margin → loss near zero. Negative margin → large loss.
Where it comes from, in four moves. (1) The KL-constrained RLHF objective max_π E[r(x,y)] − β·KL(π‖π_ref) has a closed-form optimum: π*(y|x) = (1/Z(x))·π_ref(y|x)·exp(r(x,y)/β) — the reference, tilted toward reward. (2) Invert it: r(x,y) = β·log(π*(y|x)/π_ref(y|x)) + β·log Z(x). The partition function Z(x) is an intractable sum over all responses — but it depends only on the prompt. (3) Substitute into the Bradley–Terry preference model P(y⁺ ≻ y⁻|x) = σ(r(x,y⁺) − r(x,y⁻)): the two β·log Z(x) terms cancel in the difference. (4) Maximum likelihood on the preference pairs is now expressible purely through π_θ and π_ref — the loss above. Fitting it is fitting a Bradley–Terry reward model whose reward is parameterized by the policy itself.
β’s meaning is inherited, not new. It is the KL coefficient from the RLHF objective. High β makes drift from the reference expensive, so the tuned policy stays close; low β loosens the leash and lets preferences pull the policy further. The Reinforcement learning from human feedback (RLHF) module’s intuition transfers unchanged.
Gradient intuition. Differentiating gives each example a weight of σ(−margin) — the probability the current implicit reward gets the pair wrong. Confidently-correct pairs contribute almost nothing; misranked pairs dominate the update, which pushes up log-probability on y⁺ and down on y⁻.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.