Scaling the Annotator

Lesson 1 of 3 in AI Feedback: RLAIF and Constitutional Methods.

Everything in the RLHF pipeline scales with compute except one stage: the labels. Every comparison in the preference dataset began as a person reading two responses and picking the better one. InstructGPT-style pipelines (Ouyang et al. 2022, arXiv:2203.02155) were built on exactly this kind of human comparison data — and the people are the constraint. Human annotation is slow (a thoughtful comparison takes minutes, and you need a lot of them), expensive (trained annotators, detailed guidelines, quality control on top), and inconsistent (two annotators reading the same pair disagree with each other — and with themselves on a different day). Harmlessness labeling adds a human cost on top of the economic one: someone has to read the disturbing content to label it.

So the label supply cannot simply be turned up like a compute budget. And post-training keeps wanting more labels: more capabilities to cover, more languages, more edge cases, new models to retrain. That pressure produced an obvious-in-hindsight idea: if a capable model can follow instructions, it can follow annotation guidelines. Write the rubric down, hand it to a model, and let the model label response pairs. Reinforcement learning from AI feedback — RLAIF — is that move. The term entered the literature with Anthropic’s Constitutional AI paper (Bai et al. 2022, arXiv:2212.08073), which we take apart in the next lesson.

Two ways to label the same comparison

  1. Prompts + sampled response pairs

    Identical starting point for both pipelines: the policy model generates two candidate responses per prompt.

  2. Human annotators compare

    People apply written guidelines. High-signal but slow and costly; throughput is capped by hiring, training, and reviewer well-being.

  3. Judge model applies a rubric

    A capable model is prompted with the same kind of guideline text and picks the better response. Cheap and fast; quality is bounded by the judge and the rubric.

  4. Preference dataset

    From here on the two pipelines are indistinguishable — a pile of (prompt, chosen, rejected) records.

  5. Train reward model

    Same reward-model training either way; the reward model cannot tell who produced its labels.

  6. Optimize the policy

    RL against the reward model with a KL penalty, exactly as in RLHF.

Notice what the swap does not change. Downstream of the labels, the machinery is untouched: the same Reward model training, the same policy optimization with a KL penalty anchor, the same exposure to Reward hacking if the reward signal has exploitable gaps. RLAIF is not a new alignment algorithm — it is a new label source feeding the algorithm you already know. That is exactly what makes it attractive (everything you built for RLHF still works) and exactly what makes it risky (every flaw in the judge flows straight through the pipeline into the model’s behavior, at machine speed and machine scale).

What changes is the economics and the error profile. A judge model labels comparisons for the cost of inference, around the clock, with perfect consistency in the sense that the same input yields the same distribution of judgments. Preference data stops being a scarce, hand-made resource and becomes something you can generate in bulk — including for topics where human annotators are hardest to recruit or most burdened. But the labels are now only as good as the judge’s ability to read the rubric, read the responses, and apply one to the other. A rubric the judge misreads is misread millions of times, identically.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.