The eval pyramid: deterministic first, judges last

Lesson 1 of 5 in Regression Suites in CI: Evals That Run On You.

You have a golden dataset and a pass rate. Now you need the thing that makes them matter: a suite that runs without anyone deciding to run it, on every change, with the authority to stop a merge.

The trap on the way there is running the wrong checks at the wrong cadence. A suite where every case is graded by an LLM judge costs money per run, takes minutes, and has its own error bars — so it gets run weekly, then monthly, then never. A suite that only checks whether the JSON parses runs in four seconds and catches nothing interesting. Both fail the same way: they stop being consulted before decisions.

The fix is a pyramid. Sort your checks by cost and reliability, not by how sophisticated they feel. Cheap, deterministic checks form a wide base that runs on every commit. Expensive, noisy grading sits at the apex and runs on a schedule or at a release gate. Every level catches a distinct class of bug, and the levels below a failure tell you where to look first.

The agent eval pyramid — climb only when the tier below is green

Judge grading — nightly or release-gated · slow, costly, noisy
An LLM judge scores what no assertion can express: was the tone right, was the explanation faithful to the retrieved source, did the plan make sense. Powerful and irreplaceable for open-ended output — and the only tier whose grader is itself non-deterministic, so its disagreement with yesterday may be the judge moving, not the agent.

Cost per case is a full extra model call (often a bigger model), latency is seconds, and the score has variance you must measure before you can trust a 3-point drop. Run it nightly across the whole suite, and as a gate on release candidates — not on every push.
Assertions on outputs — every commit · cheap once the run exists
Deterministic predicates over the final answer and the final world state: the reply contains the order id, the quoted price matches the ERP, the ticket ended in the right queue, no email address appears in the summary, the database row was actually written.

These are ordinary test assertions that happen to run against a probabilistic system. They are the highest-value tier per engineering hour: unambiguous verdicts, no grader to calibrate, and they encode the things a customer would actually complain about.
Tool-call correctness — every commit · reads the trajectory, not the prose
Assert on the tool calls in the trace: was search_kb called before answering a policy question, were the arguments well-formed and in range, was the forbidden tool never called, did the agent stop after one send_email.

This is trajectory checking in its cheapest form, and it is where tool-description regressions surface first — a reworded docstring that quietly stops the model reaching for the right tool shows up here long before the final answer looks wrong. Keep the assertions loose about order unless order genuinely matters, or you will fight the model for no benefit.
Schema and structural validity — every commit · milliseconds, zero flake
The floor: does the output parse, does it satisfy the output schema, are required fields present, are enums in the allowed set, is the tool-call payload valid against the tool schema. Many of these can be checked against recorded outputs with no model call at all.

Nothing here is interesting — which is exactly why it belongs at the base. It is nearly free, it never flakes, and a break at this tier invalidates every score above it, so you fail fast and stop burning tokens on a build that cannot pass.

Read the pyramid bottom-up when you are building and top-down when you are debugging. Building: get schema and tool-call checks running on every commit first, because they are the ones that will still be running in six months. Debugging a judge-tier failure: check whether the tiers below moved. A faithfulness score that fell four points while tool-call correctness fell eleven is not a writing problem — it is a retrieval problem wearing a writing problem’s clothes.

One number to keep in view: your per-commit budget. If the fast tiers take longer than a few minutes, engineers start pushing with the suite skipped, and you are back to vibes with extra YAML.

The same four tiers, as an engineering budget
Tier	Model calls per case	Verdict	Main flake source	Where it runs
Schema / structural	Zero (replayed) or one (live run)	Binary, stable.	None worth naming.	Every commit — fail the build in seconds.
Tool-call correctness	One agent run	Binary per assertion.	Legitimate path variation — the model solves it a different, valid way.	Every commit, on a fast slice of the dataset.
Output assertions	One agent run	Binary per assertion.	Paraphrase — the answer is right but your regex is narrow.	Every commit for the cheap cases; full set nightly.
Judge grading	One agent run plus one judge call (sometimes a larger model)	A score, with its own variance.	The judge itself: prompt drift, position bias, version changes.	Nightly, and as a release gate. Never blocking a push.

Key terms: eval, golden dataset, LLM-as-judge, structured outputs, pass rate, flake budget

Interactive sorting exercise: Ten checks, three cadences. Sort by cost and determinism, not by how important the check feels — an important check at the wrong cadence gets switched off.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.