Rubric design: broken, then fixed
Lesson 2 of 5 in LLM-as-Judge: An Instrument You Calibrate, Not an Oracle You Trust.
A judge is a rubric with an LLM attached. Almost every complaint about judges — it is too lenient, it flip-flops, it disagrees with our reviewers — is a complaint about the rubric, and the rubric is the part you fully control.
Five properties separate a rubric that measures something from a prompt that generates opinions. One quality per rubric. Behavioural, concrete language rather than adjectives. Anchored grades, each with an example of what earns it. Evidence before verdict in the output. And an explicit path for cannot tell from what I was given. Here is the same eval, written badly and then written properly.
The rubric that fails
“You are an expert evaluator. Rate the helpfulness of the assistant’s answer on a scale of 1 to 10. Return only the number.”
This is the single most common judge prompt in production, and it is what teams are actually looking at when they report “our helpfulness score is 7.8, up from 7.6.”
It runs. It returns numbers. It correlates with almost nothing.
What breaks, line by line
“Helpfulness” bundles at least four qualities — factual correctness, completeness, tone, and format — into one number. When the score drops you cannot tell which one moved, and a correctness regression can be masked by a tone improvement in the same decimal.
“1 to 10” offers ten grades with zero definitions. Nothing distinguishes a 7 from an 8, so the judge invents the boundary fresh each call, and the numbers cluster in a narrow band where real regressions hide inside the noise.
“Expert evaluator” is flattery, not instruction. It tells the model nothing about what to look at.
“Return only the number” forbids reasoning, so the score arrives with no evidence you can audit and no way to see whether the judge looked at the right thing. It also removes your only defence against a bogus grade: reading the justification.
Nothing is provided to judge against. No retrieved passages, no task definition, no trace. The judge is grading plausibility from vibes, and plausibility is exactly what a fluent wrong answer maximises.
No abstain path. Forced to produce a number even when the evidence is missing, the model produces one — usually a comfortable 7.
The rubric that works
Quality being graded: groundedness. Grade nothing else — not style, not completeness, not length.
You are given a user question, a set of numbered source passages, and an assistant answer. A claim is “supported” only if a specific passage states it or directly entails it. Background knowledge you happen to have does not count as support.
Step 1 — list every factual claim in the answer as a separate line. Step 2 — for each claim, name the passage number that supports it, or write UNSUPPORTED. Step 3 — only then, assign one grade using the anchors below.
If the passages are missing or empty, return
grade: nullwithreason: "insufficient evidence". Do not guess.Output JSON:
{ claims: [{ text, support }], grade, failing_claim }.
Same eval budget, radically different instrument. It names one quality, defines support operationally, forces claim-by-claim evidence before the verdict, gives the judge somewhere to go when it cannot tell, and returns structured output you can aggregate, diff, and audit.
The failing_claim field is the sleeper feature: when the score drops next week, you have the sentence that broke, not a decimal.
| Grade | Anchor (behavioural, not adjectival) | Example, given a passage that says “the standard plan includes 3 seats” | Judge must also return |
|---|---|---|---|
3 — fully grounded | Every claim maps to a named passage. No claim relies on outside knowledge. | “The standard plan includes 3 seats [P2].” | The claim → passage mapping. |
2 — grounded with an unsupported detail | All load-bearing claims are supported; one incidental detail is not, and correcting it would not change the answer’s meaning. | “The standard plan includes 3 seats [P2], and most teams start there.” | The unsupported detail, quoted. |
1 — a load-bearing claim is unsupported | At least one claim the user would act on has no supporting passage, though nothing contradicts the sources. | “The standard plan includes 3 seats and you can add more for $10 each.” | The failing claim, quoted. |
0 — contradicts the sources | A claim conflicts with a passage, or the answer cites a passage that does not say what it is cited for. | “The standard plan includes unlimited seats [P2].” | The contradicted passage number. |
Notice the grades are ordered by what went wrong, not by how good the answer felt. That is what makes them anchors: a judge — or a human — can walk the list and stop at the first description that matches. Four grades is usually the sweet spot. Two grades (pass/fail) lose the ability to distinguish a cosmetic miss from a dangerous one; ten grades invent distinctions nobody can defend.
Notice also what is absent: nothing in the rubric mentions style, length, or how confident the answer sounds. Those qualities are not unimportant — they are separately gradable. If you care about tone, write a tone rubric and run it as a second call. One quality per rubric is not tidiness; it is the only way a moving score tells you which thing moved.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.