Judging trajectories, and diagnosing disagreement

Lesson 5 of 5 in LLM-as-Judge: An Instrument You Calibrate, Not an Oracle You Trust.

Grading a final answer is the easy case. The interesting agent failures live in the trace: the agent that edited a file it never read, retried the same broken call eleven times, or produced a correct answer by a route you would never authorise.

So teams point a judge at the trace and ask it to rate this run from 1 to 5. It is the same mistake as lesson two, amplified by scale. A trace is thousands of tokens of tool calls and outputs; asked for one global verdict, the judge latches onto surface features — did the run end confidently, was the final message well formatted — because those are the features a vague question makes salient. You get a number that tracks polish and misses the blind write entirely.

Replace one vague question with several checkable ones, each answerable from the trace with a quoted span as evidence. This is also where you discover how many of your "judgment calls" were assertions in disguise.

The prompt that fails

“Here is the full trace of an agent run. Rate the quality of the agent’s work from 1 to 5.”

Symptoms you will see within a week:

Scores cluster at 4 and rarely move, whatever ships.
Long runs score higher than short ones that solved the task in three steps.
Runs that ended in a confident summary score well even when the summary is wrong.
Nobody can say what a 4 means, so nobody argues with it — which is why it survives.

The rubric that works

Ask targeted questions, one call each, evidence required:

“Given the trace, answer only this: did the agent inspect the current state of the target file before its first modification of that file? Answer yes, no, or unclear. Quote the tool-call ids you based this on. If the trace is truncated, answer unclear — do not infer.”

“Does every factual claim in the final message appear in a tool result in this trace? List each claim with the step number that produced it, or UNSUPPORTED.”

“Did the agent change its approach after a failure, or repeat the same call with the same arguments? Quote the repeated calls if any.”

Each is a yes/no/unclear question with a defined evidence requirement, so you get a pass rate per behaviour rather than a mood ring. The unclear option is load-bearing: without it, a judge handed a truncated trace will confidently answer about the part it can see.

The parts that became assertions

Write the targeted questions down and something useful happens — several of them stop needing a model at all:

“Read before write on the same path” → compare tool-call indices in the trace. Deterministic.
“Approval event precedes the irreversible call” → scan the event sequence. Deterministic, and this one should never be a judgment call.
“No more than two identical retries” → group calls by name plus arguments hash. Deterministic.
“Finished within the step and token budget” → two counters.

What is left genuinely needs a judge: was the ordering sensible, did the agent’s stated reason match what it then did, is the final message supported by the evidence gathered. That residue is smaller than teams expect, and it is where the judge budget belongs.

Targeted trajectory questions, and which ones deserve a model
Question to ask of the trace	Failure it catches	Evidence the grader must produce	Judge or code?
Did the agent inspect the file before modifying it?	Blind writes — edits based on assumed content, the classic source of silent corruption.	Ordered tool-call ids for the read and the write on the same path.	Code. Index comparison. Never pay a model for this.
Was an approval recorded before the irreversible action?	Policy bypass — the gate skipped or fired after the fact.	The approval event and the action event, in order.	Code, and it should be enforced at runtime too, not merely evaluated.
Does every claim in the final message trace to a tool result?	Fabrication — the agent summarising what it expected rather than what it found.	Claim-by-claim mapping to step numbers, UNSUPPORTED where absent.	Judge. Claim decomposition and entailment; no rule sees a claim.
After a failure, did the agent change its approach?	Loop thrash — identical retries burning budget with no new information.	The repeated calls, plus what changed between attempts.	Both. Code catches identical retries; a judge catches “changed the arguments but not the strategy”.
Did the agent’s stated reason match the action it then took?	Rationalisation drift — the plan says “check the invoice total”, the next call refunds it.	The reasoning span and the immediately following tool call.	Judge. This is the highest-value trajectory question and has no deterministic form.
Was the step ordering a sensible route to the goal?	Detours, redundant retrieval, work done in an order that guaranteed rework.	The step list with the specific detour identified.	Judge, with anchored grades — and expect low human-human agreement until you define “sensible” per task family.

Sooner or later the calibration comes back ugly: the judge says 4.6, your two reviewers say 3.1, and someone asks which one is wrong. The instinct is to blame the model. The instinct is usually wrong — in practice the most common cause is that the judge and the humans were not looking at the same evidence.

Traces are long. Judge prompts get truncated, tool outputs get elided with “… (2,431 characters omitted)”, retrieved passages get dropped to fit a context window, and the human reviewer opens the full trace in the trace viewer and sees the part the judge never received. Before you touch the rubric, check the inputs. Then walk the rest of the list.

Your judge grades 4.6, your humans grade 3.1 — diagnose it

Interactive decision tree — outcomes:

Evidence gap — fix the inputs, not the rubric
The judge graded a different artifact than the human did. Log the exact judge prompt for every graded item, assert that no elision markers survive into it, and chunk long traces into targeted per-question calls instead of one truncated mega-prompt. This is the most common cause and the cheapest fix. Re-run calibration afterwards — the old numbers describe an instrument that no longer exists.
The rubric is the bug
Judge-human disagreement cannot be diagnosed while the humans disagree — you have no ground truth. Sit the two graders down, find the distinction they are splitting, and write it into the anchors. Then re-label a fresh slice; the items they discussed are contaminated.
Scattered disagreement — resolution, not bias
A symmetric spread usually means the scale is finer than anyone can apply consistently. Collapse to fewer, harder-anchored grades, or check whether you are grading two qualities in one call — split them and the scatter often resolves into two clean signals.
Verbosity bias, confirmed on your own data
You now have the scatter plot to prove it. Add the explicit instruction that length is not graded, require every credit to attach to quoted evidence, and re-check the length-score correlation after the change. For preference evals, report win rate bucketed by length delta so the effect cannot hide in an average.
Judge drift
The instrument moved. Re-run the frozen canary set: if canary scores shifted while the human labels did not, you have quantified the drift. Pin the judge model version explicitly, stamp version and rubric hash on every score, and re-calibrate before anyone trusts the trend line again.
Read the rationales — the rubric rewards something you did not intend
Same evidence, consistent humans, systematic generosity, no length effect, no version change: the rubric is being applied faithfully to a quality you did not mean to define. Read the judge’s own justifications on the ten worst disagreements and you will usually find the loophole in a single clause — “supported” read loosely, or a grade anchor that never mentions the failure your reviewers care about most.

Tool: Trace Debugger — Take a trace apart in the Trace Debugger and try the targeted questions yourself — including the ones that turn out to be assertions once you write them down.

Interactive flashcard deck.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.