LLM as Judge
Lesson 3 of 3 in Human Eval, Arenas, and LLM Judges.
The third answer to “no answer key” is the most tempting: let a strong model do the judging. LLM-as-judge comes in two shapes. Pointwise: hand the judge one output plus a Rubric and ask for a score per criterion. Pairwise: hand it two outputs and ask which better satisfies the rubric. Either way, the judge is just a prompted Large language model (LLM) — which means judging now scales like software. Thousands of verdicts an hour, pennies each, rerunnable on every prompt tweak; this is what makes the Grader ladder of the next module possible, and why every serious Eval harness has a judge stage in it.
The catch is documented, not hypothetical. Zheng et al. (2023) — the MT-Bench study, which validated LLM judges against human votes from Chatbot Arena among other data — reported that judges broadly can approximate human preference on chat-quality comparisons, and simultaneously catalogued systematic biases. Position bias: presented with the same pair in swapped order, the judge’s verdict can flip — it favors an answer partly for where it appears. Verbosity bias: longer answers attract higher scores beyond what their content earns. Self-enhancement (self-preference) bias: a tendency to favor outputs that resemble the judge’s own — including answers generated by the judge’s own model family (the study flags this one as harder to isolate cleanly, which is a reason for caution, not comfort). These are not random noise; they are systematic, so they push every comparison in the same direction and quietly rig leaderboards.
| Bias | What it inflates | Basic mitigation |
|---|---|---|
Position bias | Whichever answer occupies the judge’s favored slot — in a naive setup, a systematic edge to one side of every comparison | Run every pair in both orders; count only order-consistent wins, score the rest as ties (see the deep dive). Randomize order where both-orders is too costly. |
Verbosity bias | Longer, more elaborate answers — including padded ones; a candidate tuned to be wordy can outscore a better, terser one | Rubric anchoring: score named criteria against described levels instead of asking “which is better?”; instruct the judge that length is not a merit; spot-check long-vs-short verdicts with humans. |
Self-preference / self-enhancement | Outputs stylistically close to the judge’s own — worst case, the judge’s own model family in a comparison it is refereeing | Use a judge from a different family than any candidate; never let a model judge a contest it competes in without human verification of the verdicts. |
Beyond bias-specific fixes, two habits keep judge scores meaningful over time. Pin the judge: the judge model, its version, and the judging prompt are part of your measuring instrument — change any of them and scores stop being comparable with last month’s, a silent form of metric Drift. Version them like code, and when you must upgrade the judge, re-run it over a reference batch so you can splice the old and new scales. Calibrate against humans: sample the judge’s verdicts, have humans grade the same items with the lesson-one machinery, and measure agreement — judge-versus-human agreement is the number that decides how much authority the judge deserves on your data, and it needs rechecking whenever the task, the rubric, or the judge changes. The full calibration protocol — and judging multi-step agent trajectories rather than single outputs — is our sister AI Agent Academy’s territory; here we stop at model-level scoring.
Pairwise both orders: the cheapest de-biasing move
Position bias would be harmless if it were random — noise averages out. It is dangerous precisely because it is systematic: the same pair, same judge, same rubric, opposite orders, different winner. From a single presentation order you cannot see it at all; every verdict looks confident and clean.
The fix exploits the one thing you fully control: presentation. Run every comparison twice — A-then-B, then B-then-A. Now each pair yields two verdicts with three possible patterns. Consistent win (same answer wins both times): the verdict survives an order flip, so position was not the deciding factor — keep it. Contradiction (each answer wins the round where it sat in the favored slot): the judge was answering “which position?” not “which answer?” — score it a tie, because the comparison carried no reliable signal. Ties in one or both orders: weak preference; treat conservatively.
Two properties make this the best-value mitigation in the toolbox. First, the price is exactly one extra judge call per comparison — trivial next to a human round, and independent of how strong the bias is. Second, it is self-measuring: the contradiction rate across a batch is a live estimate of how position-swayed your judge-plus-rubric combination is on your data. A stable low rate means order-consistent verdicts are trustworthy; a rising rate after you change the judge model, the rubric, or the task warns you the instrument has degraded — before the corrupted scores reach a decision. Zheng et al. (2023) used order-consistency in this same spirit when quantifying position bias. Contrast that with mitigations like rubric anchoring, which improve verdicts but give you no readout of how much bias remains: both-orders is the rare fix that also audits itself.
In production
All three clouds ship LLM-as-judge as a built-in feature of their evaluation tooling: pick a judge model, a rubric or built-in metric, and a dataset, and the platform loops the judge across outputs. Convenient — and it means the biases of this lesson ship inside the product, wearing an official dashboard.
AWS
Amazon Bedrock’s model-evaluation tooling includes an LLM-as-a-judge mode: a judge model you select scores outputs against built-in or custom metrics, alongside programmatic metrics and human-review workflows. The judge’s verdicts inherit judge biases regardless of who operates the loop, so use the human-evaluation path for a sample of the same items and check agreement on your own data before treating dashboard scores as ground truth.
Azure
Azure AI Foundry’s evaluation stack offers AI-assisted evaluators — relevance, coherence, groundedness, and similar quality metrics are produced by a prompted judge model, not by a formula. Treat each evaluator plus its underlying model version as part of your measuring instrument: pin them so numbers stay comparable across runs, and validate against a human-labeled sample from your own traffic rather than assuming a built-in metric is calibrated for your task.
Google Cloud
Vertex AI’s generative-AI evaluation service supports pointwise scoring and pairwise comparison, where an autorater model picks the better of two responses against criteria you define. Pairwise autorating inherits position bias like any judge, so check verdict consistency across orders, and spot-check autorater preferences against human judgment on your prompts before letting a side-by-side report pick your production model.
Key terms: LLM-as-judge, Judge bias, Rubric, Grader, Arena, Elo rating
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.