The grading ladder: deterministic first, judges last

Lesson 4 of 6 in Eval Fundamentals: You Cannot Improve What You Cannot Measure.

Now the mechanical question: for a given behaviour, how do you decide pass or fail? There is a ladder, and the rule is to stay as low on it as the behaviour allows.

The bottom rungs are code. They are free, instant, perfectly repeatable, and they never argue. The top rungs are people. They are slow, expensive, and irreplaceable for genuine judgement. Between them sits the LLM judge — powerful and fashionable, and the single most over-used tool in the evals toolbox, because a judge is itself a non-deterministic component that needs its own validation. Every check you promote from code to judge is a check whose reliability you now have to prove.

The grading ladder — climb only when the rung below cannot reach

  1. Human expert review — scarce, authoritative, the source of ground truth

    Reserved for cases where correctness is genuinely a matter of expert judgement — a legal nuance, a clinical caveat, a tone call in a sensitive complaint — and for calibrating everything below: you cannot know whether your judge is any good without a set of human-labelled cases to compare it against. Budget humans for sampling and calibration, not for volume.

  2. LLM-as-judge with a rubric — for qualities rules cannot express

    A model grades the output against an explicit rubric. Use it where the property is real but not expressible in code: faithfulness to sources, tone, whether an explanation actually answers the question. A judge is a component with an error rate — it needs its own agreement measurement against human labels, and its prompt needs version control like any other prompt. Full treatment in the LLM-as-judge module.

  3. Heuristic and fuzzy checks — still code, tolerant of surface variation

    Deterministic code that does not demand character-exact output: numeric tolerance, set overlap on extracted entities, regex for a required citation format, embedding similarity above a threshold. Cheap, repeatable, and it absorbs the harmless variation that makes exact match brittle on free text.

  4. Assertions on the trajectory — code reading the span tree

    The trajectory rules from the previous lesson, expressed as code over the trace: required tool called, forbidden tool absent, no duplicate side-effecting call, step count bounded, stop reason as expected, approval gate present with a recorded decision. Free to run, impossible to argue with, and the highest-value rung most teams have not built.

  5. Deterministic output checks — start here, always

    Exact match, schema validation, type and range checks, does the resulting database row equal the expected row. If a behaviour can be graded here, grading it anywhere else is a self-inflicted wound: you pay tokens, add latency, and introduce grader noise into your own measurements.

Interactive sorting exercise: Assign each check to the cheapest grading method that can actually decide it. Over-promoting to a judge costs money and adds grader noise; under-promoting produces a check that is confidently wrong.

The classifier above gives you the intuition. Here is the decision procedure to run on any new behaviour someone asks you to “add an eval for”. Note where it lands most of the time: further down the ladder than the person asking expected.

How should this behaviour be graded?

Interactive decision tree — outcomes:

  • Deterministic outcome check

    Exact match, schema validation, or a database assertion. Free, instant, zero grader noise. If you find yourself reaching for a judge on a behaviour that lands here, the real problem is usually that your agent does not emit a structured result you can assert on — fix that instead.

  • Hard trajectory assertion — and a runtime control

    Assert it in code over the span tree, treat a violation as a failed run regardless of outcome, and enforce it in the runtime with scoped tools or an approval gate. An eval tells you the invariant broke; only the runtime stops it breaking. Never delegate an invariant to a probabilistic grader.

  • Soft trajectory metric — track, do not gate

    Record step count, duplicate-call count, token and latency cost, then watch the distribution across releases. Alert on drift, not on a single run: failing a run for taking nine steps instead of seven produces a flaky suite and teaches the team to ignore it. Cost and latency budgets get their own treatment in the cost module.

  • Fuzzy deterministic check

    Still code: numeric tolerance, required-entity overlap, citation-format regex, similarity threshold. Measure its false-positive rate on a handful of known-good and known-bad outputs before you trust it — a threshold nobody validated is a judge with worse judgement and better marketing.

  • LLM judge with a versioned rubric

    Legitimate use. Requirements, not optional extras: a written rubric with anchor examples, a human-labelled calibration slice to measure agreement against, the rubric under version control, and awareness that changing the judge model changes your metric. Do not let the judge grade its own agent output style — and do not let a judge decide an invariant.

  • Human review — and no metric yet

    If qualified humans disagree, there is no ground truth for any grader to approximate. Two honest moves: sharpen the definition until they agree (usually possible, and it improves the product spec too), or convert the case into checkable properties — cited a source, escalated, refused — and stop pretending the fuzzy part is measured.

Tool: Eval Suite Builder — Build a suite for a real agent in the Eval Builder: pick cases, choose a grading method per behaviour, then watch what your choices do to the pass rate and to your token bill.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.