The Limits of Self-Supervision
Lesson 3 of 3 in AI Feedback: RLAIF and Constitutional Methods.
A model grading a model has an obvious recursive smell, and the smell is pointing at something real. Three families of risk deserve a permanent place in how you think about AI feedback.
Shared blind spots. Judge and student are usually siblings — same architecture family, overlapping training data, often literally the same Base model lineage. Where the student is confidently wrong, the judge is disproportionately likely to be confidently wrong in the same way. A pool of human annotators makes errors too, but their errors are largely idiosyncratic and partly cancel; a judge model’s errors are correlated with exactly the errors it is supposed to catch. A plausible-sounding Hallucination that fools the policy’s own training distribution tends to fool the judge drawn from the same distribution.
Bias amplification. Judge models have measurable preference quirks — the well-documented ones include favoring longer answers, confident tone, flattery, and particular formats. Optimize a policy against a quirky judge and the quirk compounds: each training round makes the policy more verbose, more assured, more agreeable, because that is what scores. This is Reward hacking wearing a new coat — Goodhart’s proverb again: when the measure becomes the target, it stops being a good measure. The measure here is another model’s taste.
Drift from human intent. The rubric is a proxy for what people want; the judge’s reading of the rubric is a proxy for the rubric. Optimize hard against a proxy of a proxy, generation after generation, with no fresh human signal, and behavior can satisfy every written principle while sliding away from what the principles were for. Nothing in the loop pulls it back.
| Failure mode | Why it happens | How it shows up | Standard mitigations |
|---|---|---|---|
Shared blind spots | Judge and policy share lineage and data, so their errors correlate — the judge misses precisely the mistakes the policy tends to make | Offline scores look great while plausible-but-wrong answers sail through; error rates cluster in the same domains for judge and policy | Human audits on samples; judges from a different model family; verifiable checks (tests, execution, retrieval) wherever ground truth exists |
Bias amplification | The judge’s stylistic preferences — length, confidence, flattery, format — become the optimization target and compound each round | Answers grow longer and more assured without growing more correct; sycophancy rises; one house style hardens | Bias probes (position swaps, length-controlled comparisons); mixing human labels back in; targeted evals for sycophancy and verbosity |
Drift from human intent | Optimizing a proxy of a proxy: behavior satisfies the letter of the rubric while sliding from its purpose, with nothing to pull it back | Written principles all pass, yet human raters grow less satisfied; regressions surface in areas no principle anticipated | Periodic fresh human preference collection; constitution revisions when gaps are found; red-teaming; holdout human evals as the final arbiter |
This is why the frontier has not gone human-free, and — as far as public reports let us see — is not trying to. Published pipelines describe hybrids: AI feedback carrying the bulk volume, humans concentrated where they are irreplaceable — setting and revising the rubric, auditing samples, judging the highest-stakes categories, red-teaming, and serving as the holdout evaluation that AI-optimized metrics cannot be allowed to grade. The honest summary is that AI feedback changed where human judgment sits in the pipeline, not whether it is needed. It moved up the stack: from labeling every example to writing the rules, checking the checker, and owning the final word on quality.
In production
AI feedback is a large part of why alignment data could scale to what modern assistants needed — and it leaves a fingerprint you may have noticed: assistants from different vendors often share stylistic quirks (hedging formulas, bullet-heavy structure, similar refusal phrasing). One hedged, plausible explanation is that model-generated training data and model judges circulate similar preferences through the whole ecosystem. For you as a consumer of post-trained models, the practical consequences are the same on every cloud.
AWS
Amazon Bedrock hosts models from multiple providers, and each arrives with its own post-training regime — different rubrics, different feedback mixes, different judges — which you inherit sight unseen. That is why guardrail layers exist as a separate, configurable mechanism on top of the model: the provider’s trained-in preferences do not know your acceptable-use policy. Treat a model switch as a behavior-profile switch, and re-run your own evaluations rather than assuming alignment transfers.
Azure
Azure AI Foundry pairs its model catalog with platform-level content filtering that operates independently of whatever alignment each model was trained with. The layering is the lesson: trained-in behavior is a distribution shaped by someone else’s feedback pipeline, so the platform gives you a second, auditable control point you configure to your own policy — and the two layers fail differently, which is exactly why both exist.
Google Cloud
Vertex AI’s model garden plus configurable safety filters follow the same pattern: the model brings its vendor’s post-training, the platform brings policy controls you own. If you evaluate candidate models with your own LLM-as-judge pipeline — a common pattern — remember that every caveat from this lesson now applies to your judge: shared blind spots and stylistic bias do not care whether the judging happens in training or in procurement.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.