The Grader Ladder

Lesson 2 of 4 in Building an Eval Harness.

A golden set gives you items; every item now needs a verdict. The component that produces verdicts is the Grader, and choosing one is the most consequential design decision in the harness — it determines what your scores mean, what they cost, and how much you can trust a delta.

Graders form a ladder. At the bottom: deterministic string comparison — free, instant, perfectly repeatable, and blind to meaning. At the top: an LLM judge reading each output against a Rubric — flexible enough to grade anything, at the price of tokens, latency, and biases of its own. Every rung in between trades some determinism for some flexibility. The craft is climbing only as high as the task forces you to.

The grader ladder — deterministic base, judge apex

  1. LLM judge with a rubric — Open-ended quality — the rung of last resort

    A judge model scores each output against your written criteria. It can grade what no code can — helpfulness, faithfulness, tone — but each verdict costs tokens, repeats imperfectly, and inherits documented judge biases. Use it only for criteria the lower rungs cannot express, and spot-check it against human labels.

  2. Programmatic checks — Code can decide

    Run the output and let code judge it: execute generated code against unit tests — the HumanEval / pass@k mechanism — validate JSON against a schema, check that a SQL query returns the expected rows, verify cited IDs exist. Deterministic and meaningful, but only for criteria expressible as code.

  3. Normalized match & regex — Same answer, many spellings

    Lowercase, strip punctuation and whitespace, extract the final number or letter choice with a pattern, then compare. Absorbs harmless surface variation while staying deterministic. The trap: extraction patterns silently fail on unexpected formats — log what the regex actually captured.

  4. Exact match — The deterministic base

    String equality against a reference answer. Free, instant, perfectly repeatable — and brittle: any legitimate rephrasing counts as failure. Ideal when the answer space is genuinely closed; misleading anywhere else.

Each rung up buys expressiveness and pays for it twice: in money — a judge reads every output, so grading cost scales with items × runs × judge tokens — and in reliability — the judge is itself a sampled model whose verdicts wobble and skew. Zheng et al. (2023) measured the skews that matter: judges favor the first answer shown (position bias), favor longer answers (verbosity bias), and favor outputs resembling their own style (self-preference). Rubrics, randomized ordering, and spot-checks against human labels contain these; the deeper judge-calibration protocols — agreement metrics, calibration sets, judging agent trajectories — are the sister AI Agent Academy’s territory.

Real harnesses mix rungs per criterion, not per project. A structured-extraction task might gate on schema validation (programmatic), compare extracted fields exactly (exact match), and send only the free-text rationale to a judge. The lower rungs also act as tripwires around the judge: if the output does not even parse, no expensive verdict is needed.

A judge rubric template
You are grading one output from an automated system. Judge only against the rubric below. Do not reward length, style, or confidence.

## Task
{{TASK_DESCRIPTION}}

## Input given to the system
{{INPUT}}

## Reference answer (may be partial — the candidate need not match it word for word)
{{REFERENCE_ANSWER}}

## Candidate output to grade
{{CANDIDATE_OUTPUT}}

## Rubric — score each criterion 0 or 1
1. {{CRITERION_1}}
2. {{CRITERION_2}}
3. {{CRITERION_3}}

## Output format
Return JSON only:
{"criterion_scores": [0 or 1, 0 or 1, 0 or 1], "total": 0-3, "justification": "one sentence per criterion"}

Binary per-criterion scores beat 1–10 scales: judges apply “0 or 1 with a written definition” far more consistently than fine-grained numbers. The criteria are your golden-set labeling rules, reused. Pin the judge model and its prompt like any other harness component — and spot-check its verdicts against human labels before trusting them.

Key terms: Eval harness, Golden set, Grader, Rubric, LLM-as-judge, Regression testing

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.