Variants and Trade-offs
Lesson 3 of 3 in DPO and the Direct Methods.
Once the trick was visible — collapse reward modeling and policy optimization into one loss on offline pairs — variants followed fast, each attacking a different weakness of the original.
IPO (Azar et al. 2023, arXiv:2310.12036) targets overfitting. DPO’s log-sigmoid keeps rewarding ever-larger margins, so on cleanly separated pairs the policy can push the chosen/rejected gap without bound, drifting far from the reference in the process. IPO swaps the log-sigmoid for a squared-error term aimed at a fixed target margin: once a pair is separated enough, optimization stops pushing.
KTO (Ethayarajh et al. 2024, arXiv:2402.01306) targets the data bottleneck. Pairs are expensive — two responses per prompt plus a human comparison. KTO learns from unpaired binary signals instead: this response was desirable, that one was not — exactly the shape of the thumbs-up/thumbs-down telemetry a deployed product already collects. Its loss weighs gains and losses asymmetrically, a design borrowed from Kahneman–Tversky prospect theory, hence the name.
ORPO (Hong, Lee & Thorne 2024, arXiv:2403.07691) removes the last extra model. It folds an odds-ratio preference penalty directly into the SFT loss — one stage, one model in memory, no frozen reference at all.
The honest summary of the comparison literature: this is a family, not a ladder. Published head-to-heads disagree about which member wins, and the answer moves with data quality, scale, and tuning effort. What is stable is the axis they all share — offline, direct, cheap — and the trade that entire axis makes, which is the rest of this lesson.
Key terms: Direct preference optimization (DPO), Preference data, Reward model, KL penalty, Reward hacking
| Method | Data it consumes | Pipeline complexity | Characteristic failure mode |
|---|---|---|---|
RLHF (PPO) | Preference pairs → reward model; then fresh policy samples, scored online | Highest — up to four networks live (policy, reference, reward, value) plus generation inside the loop | Reward hacking: the policy finds and exploits the reward model’s blind spots (Gao et al. 2022, arXiv:2210.10760) |
DPO | Offline preference pairs (prompt, chosen, rejected) | Low — policy + frozen reference, supervised-style loop | Overfits the offline pairs; margins can grow without bound on easily separated data; learns nothing beyond what the pairs contain |
IPO | Same offline pairs | Low — same shape as DPO | Bounded margins blunt the overfitting, but it is still offline: biased pairs in, biased policy out |
KTO | Unpaired binary labels — desirable / undesirable responses | Low — same loop shape, and no pairing step during data collection | Weaker signal per example than a true comparison; quality hinges on how the binary labels were assigned |
ORPO | Preference pairs, consumed during the SFT stage itself | Lowest — one model, one stage, no reference model | No explicit KL anchor; a single loss must balance imitation against preference pressure |
Every row below RLHF in that table shares one word the cells cannot fully convey: offline. Direct methods learn only from responses that already exist in the dataset — responses some other policy produced, often the SFT checkpoint, sometimes a different model entirely. Two consequences follow.
Distribution shift. As training moves the policy, the pairs keep describing a model it no longer is. The data never contains the current policy’s actual mistakes, so the signal drifts off-target in exactly the way a fixed dataset always does. No exploration. An offline method can only reweight behaviors present in the data; it cannot discover a strategy no annotator ever wrote down. Online methods pay the sampling bill precisely to buy these two things back: fresh samples keep the signal on-distribution, and exploration lets the policy exceed its dataset. A middle path — often called iterative or online DPO — regenerates pairs from the current policy every round and relabels them (frequently with a judge model), recovering part of the online benefit at part of the online cost.
And one setting tilts decisively back to online RL: verifiable rewards. When a checker can score any fresh sample — unit tests for code, exact answers for math — there are no preference pairs to consume; the signal exists only for what the policy generates. That regime, where RL never left, is the next module.
Interactive sorting exercise: A team describes its preference-tuning situation. Which family fits better as a starting point?
In production
Direct methods are the reason preference tuning became a commodity feature of managed fine-tuning services rather than a research project. A DPO-shaped job needs exactly what supervised fine-tuning needs — a dataset in, tuned weights out — with no reward model to host and no sampling fleet to orchestrate, so the same managed pipeline can serve both.
AWS
AWS model customization (Amazon Bedrock customization jobs, SageMaker training jobs) is built around the dataset-in, weights-out job shape, and a direct-method run fits it exactly: preference records upload like any SFT dataset and the service runs an ordinary training loop. The mechanism to check on any managed preference-tuning option is its input format — if it asks only for prompt/chosen/rejected records or thumbs-style labels, it is a direct offline method underneath, and it inherits the family’s limits: no exploration, and alignment only to what your pairs contain.
Azure
Azure AI Foundry exposes fine-tuning as the same bounded job shape, and preference-format datasets slot in the same way. The operational consequence of direct methods here is cost predictability: a DPO-style run is a bounded number of passes over a fixed dataset, so it can be budgeted and scheduled like SFT — unlike an online RL loop, whose generation-inside-training makes cost depend on how the policy samples. That predictability is a large part of why managed platforms productized this family first.
Google Cloud
Vertex AI tuning follows the same pattern: a managed job consumes an uploaded dataset and returns tuned weights. The direct-methods lens tells you what any such offering did and did not do for you — it aligned the model to your recorded pairs, not to live interaction with your users — so evaluating the tuned model on real traffic afterward remains your job, whichever cloud ran the training.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.