The five biases, each with its mitigation

Lesson 3 of 5 in LLM-as-Judge: An Instrument You Calibrate, Not an Oracle You Trust.

Every measuring instrument has systematic error, and the useful thing about systematic error is that it is predictable. A judge’s biases are known, named, and each one has a mitigation that costs you something specific — usually a second inference call or a slightly worse rubric ergonomics. That trade is the job.

Treat what follows as an instrument datasheet. You would not report a temperature without knowing your thermometer reads two degrees high in direct sun; do not report a judge score without knowing which of these five effects is inflating it.

Position bias — the slot changes the winner

Mechanism. In pairwise comparison (“which answer is better, A or B?”) judges systematically favour one position. Swap the two answers and a non-trivial share of verdicts flip, which means part of your reported win rate is an artifact of ordering. The same effect shows up when several candidates, or several trace steps, are listed for comparison.

Mitigation. Run every comparison both ways and count only the verdicts that survive the swap. Report the flip rate as a first-class judge health metric — a rising flip rate means your instrument is losing discrimination on that task, usually because the candidates have become genuinely close.

Cost. Exactly 2× inference on pairwise evals. Cheaper alternative: use single-answer grading against an anchored rubric, which has no slots to bias — this is one reason rubric scoring often beats preference comparison for regression testing.

Verbosity bias — length reads as quality

Mechanism. Longer, more elaborately hedged, more confidently formatted answers score higher at equal substance. Judges reward the appearance of thoroughness, which is precisely the failure mode of a fluent wrong answer.

Mitigation. Three layers. Measure it first: plot judge score against answer length on your own data — if the correlation is strong and your rubric never mentions length, the instrument is partly a ruler. Then state in the rubric that length is not a quality being graded, and require that any credit be attached to a quoted piece of evidence, so padding earns nothing. For preference evals, compare length-matched pairs where you can, or report win rate bucketed by length delta so you can see the effect instead of averaging over it.

Cost. One extra scatter plot and a habit. This is the cheapest bias to detect and the most commonly ignored.

Self-preference — models like their own voice

Mechanism. A model tends to grade its own outputs, and outputs in its own family’s style, more favourably. When the generator and the judge are the same model, part of your quality score is the model recognising itself.

Mitigation. Use a different model family for the judge than for the system under test, and never let the same model-and-prompt combination both generate and gate a release. The strong diagnostic: re-run the eval with a second judge from another family. If the ranking of your candidates flips, you were measuring style affinity, not quality — go back to the rubric. If the ranking holds and only the absolute numbers shift, you have a usable instrument with an offset.

Cost. A second provider relationship and a second set of rate limits. Worth it for anything that gates.

Anchoring and contamination — what you show the judge changes the grade

Mechanism. Judges are dragged by anything score-shaped in their context: a reference answer presented as ideal, the previous version’s grade, the agent’s own self-assessment (“I am confident this is correct”), a model name (“candidate: production-v3”), or the grades of earlier items in the same call.

Mitigation. Blind the judge. Strip model names, version ids and self-critiques from what you paste in. Grade one item per call with a fresh context so item three cannot anchor on item two. If you supply a reference answer, label it as one acceptable answer rather than the correct answer, or the judge will penalise valid alternatives. And keep the grade last in the output schema, after the evidence.

Cost. More calls and a slightly more careful harness — one item per call also happens to make your judge trivially parallel.

Leniency — the bias that hides all the others

Mechanism. Judges are agreeable. Absent a reason to deduct, they drift to the top of the scale, so a suite of judge evals can sit at 4.6 out of 5 for a year while real quality moves underneath it. Leniency is what makes the other four biases hard to spot: everything scores well, so nothing looks broken.

Mitigation. Force the judge to name the defect — no grade below the maximum without a quoted failing claim, and no maximum grade without a complete claim-to-evidence mapping. Watch the score histogram, not just the mean. Seed your eval set with a handful of known-bad cases whose correct grade is the bottom of the scale: if the judge does not fail them, the number it reports on real traffic means nothing.

Cost. You must maintain deliberately bad examples, and resist the urge to delete them when they make the dashboard ugly. They are the negative controls; the ugly number is the point.

Your judge grades untrusted text — defend it like any other model in the pipeline

The material a judge reads is agent output, retrieved documents and tool results: content an attacker may influence. That makes the judge a prompt-injection target, and the goal is obvious — text inside the graded content that addresses the grader instead of the user, trying to talk the rubric into a passing grade. The same trick works on a judge that reads a trace full of third-party tool output.

Containment, in the order it pays off. Put the graded material in a clearly delimited data section and tell the judge that everything inside it is evidence to be assessed, never instructions to follow. Give the judge no tools, no network, no write access — a judge only needs to read and emit a grade, so a compromised judge should be able to do nothing but lie. Validate its output against a schema and reject anything out of range or unparsable rather than coercing it. Alert on grades that jumped while the underlying content contains grader-directed language. Sample-audit top grades by hand, because injection aims at the top of the scale where nobody looks. And structurally: a judge score feeds a gate, it is never the gate — a merge, a payout, or a customer-facing claim should always have a deterministic check or a human in the path.

Same principle as everywhere else in this academy: the judge’s prose instruction is a request, the harness around it is the mechanism.

Tool: Eval Suite Builder — Build a groundedness judge in the Eval Builder: write the rubric, run it against pre-labelled cases, and watch the flip rate and score histogram react as you swap orders and pad the answers.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.