Calibration: the number means nothing until you check it

Lesson 4 of 5 in LLM-as-Judge: An Instrument You Calibrate, Not an Oracle You Trust.

This is the lesson that separates teams whose judge numbers survive scrutiny from teams who get asked “how do you know?” in a design review and have nothing to say.

Calibration means comparing judge grades to human grades on the same items, before you let the judge score anything you care about. It is not a research luxury; it is the step that converts a plausible-looking number into evidence. Skip it and you are not measuring quality — you are measuring the opinion of an unexamined instrument, then making release decisions with it.

The protocol is small enough to run in an afternoon. Pull 50 to 100 real items from your golden dataset and production traffic, deliberately stratified — some clean passes, some known failures, some near the boundary, because a random sample from healthy traffic is nearly all easy cases and easy cases hide disagreement. Have two people grade them blind using nothing but the rubric. Compare the humans to each other first: their agreement is the ceiling on anything the judge can achieve, and if it is low the rubric is the bug. Only then run the judge on the identical items and compare.

The calibration loop

Rubric v1 written
One quality, anchored grades, evidence before verdict — lesson two.
Sample 50–100 real items, stratified
Cover the score range on purpose: clean passes, known failures, boundary cases. A uniform sample of healthy traffic is mostly easy items and will flatter the judge.
Two humans grade blind
Rubric only — no judge scores, no model names, no access to each other’s grades. This is your ground truth and your ceiling.
Humans agree with each other?
Measure it explicitly. Disagreement here is a rubric defect, never a judge defect.
Sharpen the anchors
Add the distinction the two graders argued about. Re-grade a fresh slice — do not reuse the discussed items, they are contaminated.
Run the judge on the same items
Identical inputs, same rubric text, judge model version pinned and recorded.
Judge–human agreement meets your bar?
Pick the bar before you see the number, and split it by direction: false passes and false fails rarely cost the same.
Read every disagreement
Each disagreement is a bug report on the rubric, the judge prompt, or the evidence you passed in. Lesson five walks the diagnosis.
Promote the judge
Pin the judge model version, version the rubric alongside code, and freeze this labelled set as a standing canary.
Rubric, judge model, or traffic mix changed?
Any of the three invalidates the last calibration. Re-run the canary at minimum; re-label if the traffic mix moved.
Gate on it — with a standing audit sample
Sample-audit graded items by hand on a schedule. Calibration is a subscription, not a purchase.

Now the measurement itself. “The judge agreed 92% of the time” is the number teams quote, and on skewed data it is close to meaningless: if 90% of your items pass, a judge that says pass unconditionally scores 90% agreement while detecting nothing. Chance-corrected agreement — Cohen’s kappa — asks how much better than that lucky baseline you actually did.

Judge calibration metrics — what each one hides
Metric	What it tells you	Where it misleads	How to use it
Human–human agreement	Whether the quality is defined well enough to be graded at all.	Two graders who discussed the rubric this morning agree more than two graders will next quarter.	Measure it before the judge, on blind grades. It is the ceiling — report the judge relative to it.
Raw agreement (% identical grades)	The headline number, easy to explain to anyone.	Inflated by skew: on 90%-pass data, an always-pass judge scores 90%.	Never report it alone. Always pair it with a chance-corrected measure and the class balance.
Cohen’s kappa	Agreement above what guessing at the base rate would give you.	Punishes rare-class evals harshly, and the conventional “substantial agreement” bands are conventions, not laws.	Agree your own bar with your team before you see the number, and hold it across rubric versions.
False-pass vs false-fail split	The direction of the judge’s error — the thing that actually costs you money.	A single agreement number averages the two together and hides an instrument that only ever errs generously.	Build the confusion matrix. Decide which direction you can tolerate, then tune the rubric toward that asymmetry deliberately.
Rank correlation (ordinal scales)	Whether the judge orders items the way humans do, even with an offset.	Strong correlation with a systematic offset still fails any absolute threshold you gate on.	Good for comparing candidates or tracking releases; not sufficient for “must score ≥ 3 to ship”.
Order-swap flip rate	How much of a pairwise verdict is position artifact.	Looks fine on easy pairs and explodes on the close ones you care about.	Track it per eval, not globally. A rising flip rate means the instrument has run out of discrimination.

Judge drift is a real failure mode — treat it like one

Your judge score can move while your agent does not change at all. Five triggers, all common: the judge model version changed (including a provider updating a floating alias underneath you), someone edited the rubric or the prompt template, the evidence you pass in changed (a longer trace, a truncated context, more retrieved passages), the traffic mix moved so the judge now sees cases it was never calibrated on, or the eval set grew and the new items are not distributed like the old ones.

Defence. Pin the judge model to an explicit version and record it with every score. Version the rubric in the repo and treat a rubric edit as a code change with a re-calibration attached. Keep the labelled calibration set as a frozen canary and re-run it on every judge change — if the canary score moves and the labels did not, the instrument moved. Alert on canary movement the way you alert on latency. And plot judge scores with the judge version and rubric version as annotations, so “quality improved in week 12” can be checked against “we upgraded the judge in week 12.”

A judge score with no version stamped on it is not a measurement, it is an anecdote with a decimal point.

“We do not have people to label a hundred items”

Then label twenty. A twenty-item calibration with a documented disagreement list beats an uncalibrated judge by an enormous margin, and it usually finds the rubric defect within the first five items. Stratify hard — five clean passes, five known failures, ten boundary cases — and spend the labelling budget where the judge is most likely to be wrong.

What you must not do is skip the step and gate on the number anyway. If nobody can spare an hour to label twenty items, that is a statement about how much the team believes the metric, and you should treat the judge as a monitoring signal rather than a gate.

“Our judge agrees with humans 92% of the time, so we are fine”

Ask two questions. What is the pass rate in that sample? If 90% of items pass, 92% agreement is barely above an always-pass judge — compute a chance-corrected figure. Which direction are the errors? Eight percent of items graded wrong is a very different problem if all eight are false passes on the safety eval than if they are scattered.

Then ask the awkward one: agreement measured when, against which judge model version, and has the rubric changed since?

“We will calibrate once the eval suite settles down”

The suite never settles, and meanwhile the numbers are already in slide decks. Calibration is cheapest at rubric v1, when a defect costs you a rewrite instead of six months of decisions made on a bad instrument.

There is also a compounding cost. Uncalibrated judge scores get baked into targets, dashboards and performance reviews; by the time you calibrate and discover the judge was lenient on the exact failure mode your customers complain about, the organisation has a year of “improving quality” it needs to explain.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.