AI Feedback: RLAIF and Constitutional Methods

When the annotator is a model: AI-generated preferences, constitutions and critique loops, and the honest limits of self-supervision.

Content last verified 2026-09.

Lessons

1. Scaling the Annotator
2. Constitutions and Critique Loops
3. The Limits of Self-Supervision

Sources

• Bai et al. (2022) — Constitutional AI: Harmlessness from AI Feedback
• Ouyang et al. (2022) — Training language models to follow instructions with human feedback (InstructGPT)
• Gao, Schulman & Hilton (2022) — Scaling Laws for Reward Model Overoptimization