Constitutions and Critique Loops

Lesson 2 of 3 in AI Feedback: RLAIF and Constitutional Methods.

If a model is going to apply your values, you have to write them down. Constitutional AI (Bai et al. 2022, arXiv:2212.08073) makes that literal: the training pipeline is driven by a constitution — a list of natural-language principles, each a short instruction like choosing the response that is less harmful, less deceptive, or more considerate. No single principle carries the whole load; during training, principles are drawn from the list at random, so the model’s behavior ends up shaped by the ensemble.

The paper had two motivations worth remembering. First, scale with less human exposure: harmlessness labeling forces people to read harmful content, and AI feedback can shoulder most of that. Second, a better trade-off between harmless and helpful: earlier safety training tended to produce evasive models that answer hard questions with a flat Refusal. Constitutional AI explicitly aimed for models that engage — explain why a request is problematic rather than stonewalling — and the paper reports getting harmlessness without that evasiveness, using no human harmlessness labels (human labels were still used for helpfulness).

The constitution does its work through two loops: a critique-and-revision loop that manufactures supervised training data, and a preference-generation loop that manufactures comparison labels for RL.

Key terms: Reinforcement learning from AI feedback (RLAIF), Constitutional AI, Preference data, Reward model, Alignment

The critique loop starts from provocation. Take prompts designed to elicit harmful output — red-team prompts — and let a helpful-only model (trained with RLHF for helpfulness, with no harmlessness training) answer them. The answers are often exactly as bad as you would expect. Then the model is asked to critique its own response against a randomly sampled principle from the constitution, and then to revise the response in light of the critique. Critique and revision can repeat for several rounds. The final revision — the answer as it should have been — becomes supervised fine-tuning data. The model has, with a rubric in hand, generated its own corrections.

Constitutional AI: from principle list to preference data

  1. Red-team prompt

    Prompts crafted to elicit harmful responses — the raw material for harmlessness training.

  2. Helpful-only model responds

    A model trained only for helpfulness answers — often harmfully, which is the point.

  3. Critique against a sampled principle

    The model is prompted to identify how its response violates one randomly drawn constitutional principle.

  4. Revise the response

    The model rewrites its answer to address the critique. Critique → revision can repeat for multiple rounds.

  5. Revisions become fine-tuning data

    The final revisions are used as supervised training targets — the SL-CAI phase.

  6. Sample response pairs from the fine-tuned model

    The SL-CAI model answers red-team prompts twice; the pair goes to a judge.

  7. Feedback model picks the better response per principle

    A model is asked, in multiple-choice form, which response better satisfies a sampled principle — an AI-generated preference label.

  8. Preference model → RL

    AI harmlessness labels (mixed with human helpfulness labels) train a preference model; the policy is optimized against it — the RL-CAI phase.

What a constitution buys

  • Auditable principles. The values driving training exist as a readable document. You can inspect it, argue with it, version it, and change it — something impossible when the values live implicitly in a million scattered human judgments.
  • Scalable harmlessness labels. Critiques, revisions, and preferences are generated at inference cost, with far less human exposure to disturbing content.
  • Engagement over evasion. Because principles can say explain rather than stonewall, the training data itself demonstrates thoughtful engagement with hard requests — the paper reports harmless models that are much less evasive.
  • A tighter feedback loop. Change a principle, regenerate data, retrain. Steering behavior becomes an editorial act, not a re-annotation campaign.

What it does not buy

  • Value-neutrality. People chose the principles. A constitution moves human judgment up a level — from labeling every example to writing the rules — it does not remove judgment from the pipeline.
  • Coverage. A finite list of principles cannot anticipate every situation. Behavior in the gaps falls back to whatever the model generalizes, and the gaps are found the hard way.
  • Faithful interpretation. A principle is applied as the model reads it. Where the model’s reading diverges from the authors’ intent, training optimizes the misreading — consistently.
  • A guarantee. Constitutional training shapes a distribution of behavior. It does not make violations impossible; Alignment remains a training outcome, not a property you can certify.
SL-CAI and RL-CAI: how the two phases fit together

The paper’s pipeline is two distinct training phases, each consuming a different product of the constitution.

Phase 1 — SL-CAI (supervised learning). Start from a helpful-only RLHF model. Run the critique-and-revision loop on red-team prompts: sample a response, sample a principle, critique, revise; optionally repeat with freshly sampled principles. Then fine-tune a pretrained model on the final revisions (with helpfulness data mixed in so the model keeps its general assistant ability). The purpose is not to finish the job — it is to get the model’s response distribution into the right neighborhood, so the RL phase that follows starts from a policy that already mostly behaves and needs less exploration.

Phase 2 — RL-CAI (reinforcement learning from AI feedback). Sample pairs of responses to red-team prompts from the SL-CAI model. Present each pair to a feedback model as a multiple-choice question: which response better satisfies this (randomly sampled) principle? The resulting AI-generated harmlessness comparisons are combined with human helpfulness comparisons, a preference model is trained on the mix, and the policy is optimized against it — structurally identical to RLHF, with the annotator swapped out for harmlessness only. This is the step the paper names RLAIF.

Two details reward attention. The feedback model’s judgments can be elicited with chain-of-thought — asking it to reason through the comparison before answering — which the paper explores for improving feedback quality. And the harmlessness signal is ensembled over principles: because each comparison is judged against one randomly drawn principle, no single sentence of the constitution dominates; the preference model absorbs the blend.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.