Why Preferences, Not Demonstrations

Lesson 1 of 4 in RLHF: Learning from Preferences.

Supervised fine-tuning teaches by demonstration: humans write ideal answers, the model imitates them. The previous module ended on its limits, and the sharpest one is this — the teacher’s pen is the ceiling. Imitation can make the model sound like its demonstrators, but it cannot make the model better than the answers they managed to write. And ideal answers are brutally expensive to produce: they demand domain expertise, careful writing, and time, for every single training example.

There is a second, quieter problem. Many of the qualities you actually care about — helpfulness, honesty about uncertainty, the right level of detail, appropriate refusals — are easier to recognize than to produce. Ask an annotator to write the perfect answer to a tricky medical question and you need a doctor with an hour to spare. Show the same annotator two candidate answers and ask which is better, and you get a usable judgment in seconds — one that different annotators agree on far more often than they agree on what the one true answer should look like.

That asymmetry is the founding insight of reinforcement learning from human feedback (RLHF): stop asking humans to demonstrate, and start asking them to compare.

The idea has a clean research arc, worth knowing because each step de-risked the next.

Christiano et al. (2017, arXiv:1706.03741) showed that deep RL agents — in simulated robotics and Atari — could learn behaviors from nothing but humans repeatedly picking the better of two short video clips. No hand-written reward function, no demonstrations: a simulated robot learned to do backflips because a human kept choosing the clip that looked more backflip-like. The paper established the template: fit a model of human preferences, then optimize against it.

Stiennon et al. (2020, arXiv:2009.01325) brought the template to language with summarization. Models tuned on human preference comparisons produced summaries that labelers preferred to the output of much larger models trained with supervised learning alone — and even to the human-written reference summaries. Recognition beat imitation, measurably.

Ouyang et al. (2022, arXiv:2203.02155) scaled the recipe into InstructGPT, the direct ancestor of modern assistants. The headline result: labelers preferred outputs from the 1.3-billion-parameter InstructGPT model over outputs from the 175-billion-parameter GPT-3 it was derived from — more than a hundred times smaller, yet judged more useful, because it was optimized for what humans actually chose.

Preference learning leaves the lab

  • 2022-03-01 — InstructGPT: RLHF at scale:

    Ouyang et al. applied reinforcement learning from human feedback at scale to make GPT-3 follow instructions; labelers preferred a 1.3B aligned model over the raw 175B one. It established RLHF as the standard post-training recipe that turns a text predictor into an assistant.

  • 2022-11-30 — ChatGPT launches:

    A free chat interface over an RLHF-tuned GPT-3.5 model reached an estimated 100 million users within two months. It was the moment LLMs became a consumer product, and it reset the industry’s research and deployment agenda around assistants.

  • 2023-07-18 — Llama 2: open weights go commercial:

    Meta released retrained 7B–70B models plus RLHF-tuned chat variants under a license permitting most commercial use. Open weights became a viable production choice rather than a research artifact, anchoring the open-versus-closed split that defines the current landscape.

The three-stage InstructGPT pipeline (Ouyang et al. 2022)

  1. Pretrained base model

    A Base model that completes text. It has the knowledge; it lacks the behavior.

  2. Stage 1 — collect demonstrations

    Labelers write good responses to sampled prompts. Expensive per example, so this set stays comparatively small.

  3. Supervised fine-tuning (SFT)

    Train on the demonstrations with the ordinary next-token objective. Output: a competent assistant draft — and the reference point everything later is measured against.

  4. Stage 2 — humans rank sampled outputs

    For each prompt, sample several responses from the model and have labelers rank them. Each ranking of K responses yields K·(K−1)/2 pairwise comparisons — cheap, fast judgments.

  5. Train the reward model

    Fit a model that assigns a scalar score to (prompt, response), trained so the preferred response scores higher. Lesson two.

  6. Stage 3 — RL against the reward model (PPO)

    The model generates, the reward model scores, PPO updates the weights — with a KL penalty keeping it near the SFT model. Lesson three.

  7. Preference-tuned assistant

    The behavior profile you meet in production: shaped by whoever supplied the preferences.

Read the pipeline as a division of labor. SFT gets the format right — it teaches the model what an assistant turn even looks like, cheaply enough because it needs relatively few demonstrations. The reward model captures taste — a reusable, queryable stand-in for the labelers. RL applies the taste at scale, grading millions of model-generated responses no human will ever read. The human effort concentrates where humans are most reliable: quick comparative judgments. The rest is compute.

Keep one caveat in your pocket for the whole module: every arrow in that diagram points through the reward model. If the reward model’s picture of human preference is wrong somewhere — and it always is, somewhere — stage three will find the error and exploit it. That is lesson four.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.