When a judge is the right instrument — and when it is laziness
Lesson 1 of 5 in LLM-as-Judge: An Instrument You Calibrate, Not an Oracle You Trust.
LLM-as-judge means exactly what it says: you hand a second model the thing your agent produced, plus a written rubric, and it returns a grade. It is the only practical way to score qualities no rule can see — is this answer grounded in the retrieved passages, is this reply appropriately careful with an angry customer, did this trajectory make sense as a plan.
It is also the most over-used tool in the eval toolbox. A judge is a measuring instrument built out of the same non-deterministic machinery you are trying to measure. Instruments have systematic error. Instruments drift. Nobody ships a thermometer without calibrating it, and nobody should ship a judge score to a dashboard without checking it against human grades first.
The grading ladder — climb only as high as you must
- Human expert grading — the ground truth, and the bottleneck
A qualified person reads the output and decides. This is the only tier that defines correctness rather than approximating it — which is why it is the reference every other tier is measured against. Cost: minutes per item, and expert attention is the scarcest resource you have. Reserve it for defining the standard and auditing the tiers below, not for grading every run.
- LLM judge with a calibrated rubric — flexible, cheap, systematically biased
A model applies your written rubric at scale — cents and seconds per item. Buys you coverage on qualities that resist codification, at the price of a calibration obligation that never ends. Everything in this module is the price tag.
- Code check over the output or trace — deterministic, cheap, needs a precise definition
A function you wrote returns pass or fail: the JSON validates, the answer cites at least one retrieved document id, the read tool was called before the write tool, the run stayed under nine steps. Slower to design than a judge prompt — you have to say exactly what you mean — and free to run forever afterwards.
- Exact or numeric assertion — the cheapest possible truth
String equality, numeric tolerance, a unit test that passes. Applies to a narrow slice of agent behaviour — but on that slice it is perfect, instant, and never drifts. If a quality can live here, it must live here.
The rule that keeps eval suites honest: use the lowest tier that can actually see the quality you care about. Reaching for a judge because writing the check is tedious is how teams end up paying inference costs, and inheriting bias and drift, to answer a question JSON.parse already answers for free — with a worse answer, since the judge is right 97% of the time and the parser is right always.
The reverse mistake is quieter and more expensive. Teams that own only deterministic checks end up measuring what is easy — latency, schema validity, keyword presence — and calling it quality, while the failure that actually hurts customers (a confident answer built on a passage that says the opposite) sails through every green check in CI. Sort the drill below before you write another rubric.
Interactive sorting exercise: Nine things an agent team wanted to measure last quarter. Which need a judge, which need code, and which no automated grader can settle?
Key terms: LLM-as-judge, eval, golden dataset, grounding, trace, non-determinism
Should a judge grade this quality — and may its score gate anything?
Interactive decision tree — outcomes:
- Write the check, not the rubric
Deterministic, free after the first hour, and immune to drift. Keep the judge budget for the qualities that have nowhere else to go.
- Your definition is the problem, not the judge
A judge cannot be more precise than your definition of the quality — it will happily launder your ambiguity into a confident decimal. If two humans disagree, the rubric is the bug. Fix it, re-test on humans, then come back.
- Judge it, and treat the number as a thermometer
Fine for trend-watching and triage queues: a drop means go look, not the system is 12% worse. Still sample-audit by hand, because an uncalibrated judge can trend the wrong way with total confidence.
- Judge it, and keep the calibration alive
A calibrated judge may gate — with a standing audit sample, a pinned judge model version, and a re-calibration triggered by any rubric or model change. Calibration is a subscription, not a purchase.
- Run it in shadow mode first
Log the score, show it on the dashboard labelled unvalidated, and let nothing block on it until you have agreement numbers. Gating on an uncalibrated judge means your release process is now driven by a bias you have not measured.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.