Outcome evals and trajectory evals

Lesson 3 of 6 in Eval Fundamentals: You Cannot Improve What You Cannot Measure.

Two questions, both mandatory, frequently confused:

Did it get the right answer? That is an outcome eval. Compare the end state against the expected end state: the refund amount, the JSON payload, the file contents, the row in the database.

Did it get there sanely? That is a trajectory eval. Read the trace and judge the path: which tools it called, in what order, how many times, what it did with the results, whether it stopped for the right reason.

The reason you need both is the case that breaks every outcome-only suite: the right answer reached through a forbidden action is a FAIL. An agent that produces the correct refund figure by querying the production database directly instead of calling the sanctioned pricing tool has not passed. It got lucky, in a way that will not stay lucky, using a capability you did not intend it to use.

The two axes, with what each one catches and misses
Dimension	Outcome eval	Trajectory eval
The question	Is the end state correct?	Was the path to it legitimate and efficient?
Input it reads	Final answer, side effects, resulting records.	The full span tree: tool calls, arguments, order, retries, gate decisions, stop reason.
Typical checks	Exact match on a number or id; schema validity; “does the answer cite the correct policy section”; database row equals expected.	Required tool was called; forbidden tool was not called; no duplicate side-effecting call; ≤ N steps; stop reason is `completed`, not `turn_limit`.
What it catches that the other misses	Confident nonsense delivered by a beautiful, well-formed path. A tidy trajectory that ends in the wrong number is still wrong.	Right answer, wrong means: privilege misuse, skipped gates, 40 flailing steps, a duplicate charge, a lucky guess with no supporting retrieval.
Failure it produces if used alone	You ship an agent that is correct today and unsafe structurally — the shortcut it took will eventually take it somewhere expensive.	You ship a beautifully behaved agent that is wrong. Process compliance is not correctness.
Example verdict	`refund_amount == 100.00` → pass.	`used_tool(pricing_api) && !used_tool(prod_db_read) && steps <= 8` → pass.

Trajectory rules come in two strengths, and mixing them up is a common mistake. Hard rules are invariants: never call this tool, never skip this gate, never send more than one email per ticket. They are pass/fail, they veto, and — as the foundations modules insist — they should also be enforced in the runtime, because an eval that catches a violation catches it after the fact. Soft rules are efficiency and hygiene: prefer the cache before the API, do not re-read the same file five times, keep it under ten steps. Score these as a distribution and watch the trend; do not fail a run for taking nine steps instead of seven.

The trace fields that make any of this possible are exactly the ones the previous module taught you to record: tool name, arguments, status, parent span, gate decisions, and the run-level stop reason. If your traces do not carry them, you cannot write a trajectory eval at all — you can only grade the answer and hope.

Interactive sorting exercise: Each card is one run of a refund-support agent whose expected outcome is “€100 refund, using the pricing tool, escalating anything above €100”. Grade each run on the two axes.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.