Your First Eval Suite, in a Week

You have an agent in production and no evals. Here is the five-day recipe: harvest twenty real cases from traces, write the checks a machine can settle, add outcome grading, add one trajectory check and exactly one judge, then wire it into CI with thresholds you can defend on a Friday afternoon.

You shipped the agent. It works, mostly. Support has opinions. Someone in a review asked how do you know a prompt change did not break it and the honest answer was that you tried three tickets by hand and they looked fine.

That is the state this recipe starts from: a working agent, real traffic, real complaints, and zero evals. One engineer, five days, no budget for a platform. At the end of the week you do not have an eval programme. You have something better for right now: a suite that runs in CI, fails on the things that already went wrong, and can be added to in ten minutes by whoever handles the next incident.

The running example is a refund-triage agent — it reads an inbound ticket, looks up the order, and then either drafts a reply for a human or issues a refund itself when the amount is under fifty dollars. Substitute your own agent; the shape does not change.

Here is what Friday looks like when this works:

20 cases in a directory, each one a JSON file taken from a real trace.
Roughly 40 deterministic checks across those cases — schema, tool calls, forbidden actions — that need no model to run.
One trajectory check on the one ordering that actually matters, and one judge on the one quality no rule can settle.
A CI job that runs on every prompt, tool-schema, and model change, with a threshold that is currently 18/20, not 20/20, and a written reason for each of the two failures.

The underlying material is taught in Eval Fundamentals: You Cannot Improve What You Cannot Measure (the golden dataset, outcome vs trajectory), LLM-as-Judge: An Instrument You Calibrate, Not an Oracle You Trust (rubrics and calibration), and Regression Suites in CI: Evals That Run On You (triggers, pinning, thresholds). This note is the compressed build order, not the theory.

Key terms: golden dataset, outcome eval, trajectory eval, LLM-as-judge, flake budget, regression suite

The week — build order, not a wish list

Monday 09:00 — open the trace store, not a blank test file
The single most common way this week fails is starting from imagination. Invented cases test the agent you think you built. Your traces hold the agent you actually shipped. If you have no traces, stop: spend the week on instrumentation instead and come back. Trace Anatomy: Reading an Agent Run Like a Professional is the prerequisite you skipped.
Day 1 — harvest 20 cases, every complaint included
Freeze inputs and the observed output for twenty real runs. Non-negotiable rule: every complaint from the last quarter becomes a case, even the ones you cannot grade yet. Output: a directory of JSON files and a list of the ones you could not settle.
Day 2 — deterministic checks: schema, tool calls, forbidden actions
No model in the loop. Does the output parse against the schema? Did it call lookup_order before deciding? Did it ever call issue_refund with an amount over the limit? These are cheap, fast, and they never flake. Expect roughly two checks per case by end of day.
Day 3 — outcome grading: did the run get the right answer?
Each case gets an expected outcome — refund issued, refund refused, escalated to human — plus the fields that must be right (order id, amount, reason code). Exact match where you can, normalised or numeric-tolerance match where you must.
Anything left that a rule genuinely cannot settle?
Be strict here. "The reply should be polite" is not a judge problem, it is an unspecified requirement. "The reply must not promise a delivery date the system cannot commit to" is a real judge problem: infinitely many phrasings, one rule you cannot write.
Day 4 — one trajectory check and exactly one judge
One trajectory check on the ordering that carries real consequence, and one judge with a written rubric, spot-checked against twenty of your own labels before you believe a single score. One. Not a judge per quality dimension.
Day 4 — no judge needed: add ten more cases instead
The best outcome of Wednesday is discovering you do not need a judge yet. Coverage beats sophistication in week one: thirty cases with deterministic grading is a far better suite than twenty cases with a shaky rubric bolted on.
Day 5 — wire into CI with an honest threshold
Pin the model version, run the suite on prompt/tool/model changes, and set the threshold at what the suite scores today on unchanged code. Not at 100%. A gate calibrated to a number you have never hit is a gate everyone learns to override.
Green three times in a row on unchanged main?
Three identical runs of the same commit. If the score moves, you have flake, and flake in an eval suite is not a nuisance — it is the mechanism by which the whole thing gets ignored by October.
Fix the check, not the threshold
Flake usually means an over-tight assertion (exact string match on generated prose), an unpinned dependency, or a judge with a scale nobody can apply twice. Loosen the assertion or delete it. Never fix flake by lowering the bar for real failures too.
Friday 16:00 — merges gated at 18/20, two known failures documented
The two failures are the point, not the shame. They are the first two entries on the fix list, and the suite now proves they are still broken every time anyone touches the agent.

Day 1 — the harvest

Do not write a single check today. Today you build the golden dataset, and the whole week depends on where the cases come from.

Pull twenty runs, in this proportion. Ten are the boring middle of your traffic — sample them randomly from a normal week so the suite notices when the common path breaks. Five are the tail: the longest run, the most expensive run, the one with eleven tool calls, the one that hit your step limit, the one with the empty result. Five are complaints — and if you have more than five complaints, take them all and cut the random middle instead.

Every complaint becomes a case. No exceptions, including the ones you disagree with. A complaint you decide was user error still becomes a case, with the outcome you believe is correct written down. That is how a suite accumulates institutional judgement instead of just catching crashes.

The case file is boring on purpose. A directory, one JSON file per case, checked into the same repo as the agent:

{
  "id": "refund-014",
  "source_trace": "trace_01JQ8Z4M7B2K",
  "why": "Complaint SUP-2298. Customer asked about a duplicate charge; agent refunded the wrong order.",
  "input": {
    "ticket": "I was charged twice for order 88213 on the 4th, please sort this out",
    "customer_id": "cus_4471"
  },
  "fixtures": { "orders": ["88213", "88104"] },
  "expect": {
    "outcome": "escalate_to_human",
    "must_call": ["lookup_order"],
    "must_not_call": ["issue_refund"],
    "fields": { "referenced_order_id": "88213" }
  },
  "notes": "Two orders in range. Ambiguous cases escalate — decided by Priya, 2026-08-19."
}

Four details in there earn their keep. source_trace means anyone can go read the original run instead of arguing about what the agent used to do. why stops cases from becoming unexplainable folklore in four months. fixtures is the hard part — the agent calls live systems, so you need the order lookup to return the same thing in March as it did today; record the tool results from the original trace and replay them. notes records who decided and when, because half your cases encode a judgement call, not a fact.

Cases you cannot grade yet still get files. Write the input, write "expect": { "outcome": "UNDECIDED" }, and put the disagreement in notes. You will finish Day 1 with two or three of these. They are the most valuable artifacts of the week: each one is a product decision nobody had made, now visible, with a ticket attached.

One warning about fixtures: if replaying tool results is genuinely impossible today — the tool is stateful, the sandbox does not exist — do not spend Tuesday building a fixture framework. Run those cases against a live staging system, accept that they are slower and flakier, mark them tier: "slow", and keep them out of the merge gate. Ten fast reliable cases beat twenty you cannot trust.

Six kinds of check, cheapest first. Build top-down and stop as soon as the tier above has covered the failure — every row you climb costs more and tells you less.
Check	What it catches	What it costs	How it lies to you
Schema / parse · Day 2	Malformed structured output, missing required fields, a field that turned from a number into a string after a model swap. Catches roughly a third of real breakages on its own.	Minutes. One validator call per case, no model, no network. Runs in the same millisecond budget as your unit tests.	It passes on output that is perfectly formed and completely wrong. A green schema check tells you the pipe is intact, nothing about what came through it.
Tool-call assertion · Day 2	The agent answering from memory instead of looking anything up, calling the wrong tool for the intent, or passing an argument it invented. `must_call: ["lookup_order"]` catches the single most common silent regression after a prompt edit.	Minutes per case, read straight off the trace you already emit. Zero marginal cost per run.	Presence is not correctness. It confirms `lookup_order` was called; it says nothing about whether the order id was the one the customer asked about.
Forbidden action · Day 2	The failures that actually cost money: `issue_refund` above the limit, a write to a tenant the run has no business touching, a second refund on the same order. Your incident list turns into these almost one-to-one.	Minutes. And it is the cheapest check to argue for, because every one of them maps to a sentence a lawyer or a CFO already said out loud.	It is not a control. A red eval tells you it happened in CI; only the runtime can stop it in production. Pair each one with the enforcement taught in Tool Scoping and Least Privilege.
Outcome check · Day 3	Did the run reach the right end state — refunded, refused, escalated — with the right key fields. This is the check that answers "is the product working", which is the question your manager is actually asking.	An hour or two of thinking per case, because you have to decide what right is. The decisions are the expensive part, not the code.	Exact match on generated prose is a flake factory. Compare the decision and the structured fields; never compare the sentence. And an outcome check cannot see a run that got there by a route you would fire someone for.
Trajectory check · Day 4	The route, not just the destination: eleven tool calls where two would do, a lookup loop that never terminates, an approval requested after the action instead of before. Ordering failures that outcome checks pass clean.	Half a day, plus ongoing maintenance — trajectories are the checks that break when you legitimately refactor the loop. Managed platforms offer matching modes here (Microsoft Foundry documents exact, in-order and any-order trajectory matching; verify current behaviour in the docs).	Over-specified trajectories punish improvement. Assert the constraint that matters (approval precedes payment) rather than the exact call sequence you happened to observe on Monday.
LLM judge · Day 4, once	The one quality with infinite valid phrasings and a real failure mode — for the refund agent, "the reply must not commit to anything the system cannot deliver". Nothing else in this table can see that.	A day to write and calibrate the rubric, real money and latency per run, and permanent upkeep: judge drift means the instrument needs re-checking every time you change the judge model.	It agrees with you until you stop looking. Untested rubrics collapse to a 4-or-5 scale, judges favour verbose answers and their own model family — the failure modes are catalogued in LLM-as-Judge. Treat the score as a thermometer before you let it gate anything.

Day 2 — the checks a machine can settle, and nothing else

Write the boring ones. All of them. In your existing test framework, in the same repo, run by the same command your unit tests already use. pytest and vitest are eval harnesses; you already have one.

The whole of Day 2 is a loop over the case directory:

@pytest.mark.parametrize('case', load_cases('evals/cases'), ids=lambda c: c['id'])
def test_deterministic(case):
    run = replay(case['input'], fixtures=case['fixtures'])   # returns the trace

    RefundDecision.model_validate(run.output)                # schema

    called = [c.name for c in run.tool_calls]
    for name in case['expect'].get('must_call', []):
        assert name in called, f'never called {name}'
    for name in case['expect'].get('must_not_call', []):
        assert name not in called, f'called forbidden {name}'

    for call in run.tool_calls:
        if call.name == 'issue_refund':
            assert call.args['amount'] <= 50_00, 'refund over policy limit'

That is it. Forty-odd assertions across twenty cases, no model in the loop, sub-second per case, and it will already fail on something you did not know was broken. Run it against last month’s prompt as well as today’s — the diff between the two is your first real regression report, and it is the single most persuasive artifact you will produce this week.

Day 3 — outcome grading, which is mostly deciding what right means

Today the code is trivial and the conversations are not. For each case, the expected outcome plus the fields that must match:

assert run.output.decision == case['expect']['outcome']
for field, want in case['expect'].get('fields', {}).items():
    assert getattr(run.output, field) == want

Three rules keep this from rotting. Compare decisions and structured fields, never prose — the moment you assert on a generated sentence you have bought a flake for life. Use tolerances where the domain has them: refund amount to the cent, confidence to one decimal, timestamps not at all. And when you cannot agree on the expected outcome, that is not an eval problem — mark the case UNDECIDED, escalate the product question, and move on. A suite full of contested expectations gets deleted; a suite with three open questions gets answered.

By Wednesday evening you know your real score. It is usually somewhere around 14 to 17 out of 20, and it is usually a shock. That number is the most useful thing you have learned all week.

Day 4 — one trajectory check, then earn your judge

The trajectory check goes on the ordering with consequences, expressed as a constraint rather than a transcript:

names = [c.name for c in run.tool_calls]
if 'issue_refund' in names:
    assert names.index('lookup_order') < names.index('issue_refund')
    assert 'request_approval' in names[: names.index('issue_refund')]

Then the judge — and the bar for adding one is high. It must be a quality that is real (a specific failure you have seen), unruleable (no regex, no schema, no keyword list gets there), and worth the upkeep. For the refund agent, exactly one thing qualifies: the reply must not commit the company to anything the system cannot deliver — no promised dates, no promised amounts, no goodwill credits that do not exist.

Write it as a rubric with a decision, not a five-point scale: does this reply contain a commitment (date, amount, or entitlement) that is not supported by the tool results in the trace? Yes or no, and quote the phrase. Then do the part everyone skips: label twenty replies yourself and check the judge against your labels before you believe it. If it disagrees with you on four of twenty, the rubric is broken, not the model. This is the calibration discipline from LLM-as-Judge, compressed into an afternoon.

Run the judge in report-only mode for the first week. It logs, it does not gate. That is the difference between an instrument and a superstition.

Skip: building an eval platform

This is how the week dies. Someone proposes a case registry with a database, a results API, a diff viewer, and a plugin interface for graders. Three weeks later there is an impressive amount of infrastructure and still no answer to "did the prompt change break refunds".

Instead: a directory of JSON files, your existing test runner, and CI output. Git is your case store and your version history. If the suite outgrows that — you will know, it happens somewhere north of a hundred cases — then adopt a tool rather than writing one. As of September 2026 the practical shortlist is Langfuse (MIT core, self-hostable, ships code and judge evaluators with datasets), Arize Phoenix (Elastic License 2.0, OpenTelemetry-native), and the commercial platforms LangSmith and Braintrust; if you are already on a cloud agent runtime, Amazon Bedrock AgentCore Evaluations scores agents from OTel traces and Microsoft Foundry ships built-in agent evaluators, several of them still marked preview. Check the current docs and licences before you commit — this layer moves fast.

Skip: a UI for browsing results

Nobody browses eval results in week one. They read a CI failure. Your interface is the assertion message, so spend the effort there instead: refund-014: called forbidden issue_refund (amount 4200) — expected escalate_to_human is worth more than any dashboard you could build by Friday.

Skip: synthetic case generation

Asking a model for two hundred test tickets is the most tempting shortcut available and it produces a suite that measures a fiction. Generated cases share the model’s assumptions about what tickets look like, which is exactly the blind spot you are trying to instrument.

Instead: twenty real ones. Generation becomes useful later, for variations on cases you have already grounded — same complaint, five phrasings — not for inventing the distribution.

Skip: a judge for anything a rule can settle

Every judge you add is a permanent maintenance commitment: a rubric to keep, a calibration to re-run, a model version to pin, a bill per run. One judge for one genuinely unruleable quality is a good trade. Five judges — helpfulness, tone, completeness, conciseness, safety — is a second product you now maintain, and the scores will correlate with each other rather than with anything a customer cares about.

Skip: statistical rigour you cannot afford yet

Twenty cases will not give you confidence intervals and pretending otherwise wastes Thursday. What twenty cases give you is a tripwire: a change that breaks three of them broke something real.

Do run the suite three times on unchanged code to find your flake floor. Do not build a variance-reduction framework. The non-determinism you cannot remove is a fact to be measured, not a bug to be engineered away in week one.

Skip: online evals and production sampling

Scoring live traffic, sampling production runs, shadow comparisons between model versions — all correct, all week three or later. Offline first, because an online eval you cannot reproduce is an alert with no debugger attached. Observability in Production: Watching Agents at Scale covers the online half when you get there.

Skip: chasing 100%

A suite that reports 20/20 on the day you write it is measuring nothing — you calibrated the expectations to the current behaviour. The first honest score is below full marks, and the two or three failures you ship the gate with are the beginning of your backlog, not a reason to soften the expectations until the light turns green.

Day 5 — CI, thresholds, and a flake budget in writing

Four decisions, all of them written down in the repo, all of them arguable in public.

What triggers a run. Prompt files, tool schemas, agent code, and the pinned model version. That last one is the trigger teams forget: a model upgrade is a code change, and if bumping the model id does not run your evals then your gate has a hole exactly where the risk is. Deterministic tiers run on every commit — they cost nothing. The judge tier and any tier: "slow" cases run on the pull request and on a nightly schedule, not on every push.

What is pinned. The model id with its version suffix, the judge model separately, the prompt template hash, and the tool schemas. Record all four in the results file. Without them a red run six weeks from now is unattributable, and unattributable failures get retried until they pass.

The threshold, set to the honest score you have today. Two gates, not one:

gates:
  deterministic: 20/20     # zero tolerance — these never legitimately fail
  outcome:       16/20     # the real score today; two known bugs, two undecided
  judge:         report-only
flake_budget:
  max_score_variance: 1    # across 3 runs of unchanged main
  action_if_exceeded: fix or delete the offending check — do not lower a gate

The deterministic gate is absolute because a schema violation or a forbidden refund is never acceptable. The outcome gate sits at what you actually score, with the two known failures named in a comment and each carrying a ticket. When those tickets close, the gate goes to 18, then 19. The threshold ratchets up as you fix things; it never ratchets down to accommodate a regression.

The flake budget, stated as a number. One point of variance across three runs of unchanged main is tolerable. Two is a bug in the suite. And the rule that keeps the whole thing honest: a failed eval job is never fixed by re-running it. If a check passes on the second attempt, either the check is wrong or your agent is nondeterministic in a way your users are also experiencing. Both of those are findings. Neither is resolved by a green tick on retry.

Budget the cost before you turn the gate on. Twenty cases with deterministic grading is essentially free. Add a judge and you are paying two model calls per case per run — trivial per run, and genuinely annoying at forty pull requests a day, which is why the judge tier belongs on the PR and the nightly, not the push. Regression Suites in CI: Evals That Run On You works through the pinning and tiering mechanics properly.

One thing to do before you go home Friday: put the score in a place the team already looks. A line in the PR comment, a number in the channel where deploys land. A suite nobody sees is a suite nobody defends the first time it blocks a release.

Week two, Tuesday: the suite is red. Walk it.

Interactive decision tree — outcomes:

You just taught the team that the gate is advisory
This is the failure mode that kills eval suites, and it takes about three weeks. Re-running until green means the gate now reports whether someone was in a hurry, not whether the agent works.

Fix it structurally, not culturally: make re-runs visible (log every retry against the PR), and make the flake budget an explicit number so "it is just flaky" becomes a claim someone has to substantiate. If the same job needs two attempts, that is a bug report about your suite or your agent — file it.
Your agent is nondeterministic in a way your users are feeling
A schema or tool-call assertion that flaps on identical input is not eval noise — it is production behaviour you did not know about. The same run sometimes skips the lookup. Some of your users are getting the skipped-lookup version.

Do not loosen the assertion. Go read the diverging traces side by side, find what varies (temperature, tool-result ordering, a race in a parallel call), and fix the agent. Then keep the check. Reliability Plumbing: Timeouts, Retries, Idempotency, Breakers is the toolbox.
You asserted on prose. Delete the assertion.
Comparing generated wording is the flake factory from Day 3, and it will burn your credibility faster than any real bug. Rewrite the check to compare the decision and the structured fields, or move the quality it was reaching for to the judge tier if it is genuinely unruleable.

Deleting a check that measures nothing is progress, not retreat.
The rubric is broken, not the judge
A judge that scores one run two different ways has a scale nobody can apply twice — usually a 1-to-5 rubric where 3 and 4 are indistinguishable. Collapse it to a binary decision with a required quoted justification, then re-check it against your own twenty labels.

Meanwhile: the judge tier does not gate. It never should have been gating on the strength of one afternoon of calibration.
Correct behaviour: this one blocks
A forbidden action in the deterministic tier is the case you built the suite for. It does not get a threshold, a discussion, or an override — the PR stops.

And ask the second question while you are here: if the eval caught issue_refund above the limit, what would have stopped it in production? An eval is a detector, never a control. The enforcement belongs in the runtime, as Tool Scoping and Least Privilege and Human-in-the-Loop both argue.
The suite did its job — this is the whole return on the week
Six unrelated cases going red on one change is exactly the signal that used to reach you as a customer complaint eleven days later. Revert or fix, then add the case that would have caught it earlier if none of the twenty did.

Note what just happened: a week of work turned a slow, embarrassing feedback loop into a red pipeline. That is the argument for the next week of eval work, and you should make it out loud.
You found a product decision, not a bug
Mark the case UNDECIDED, take it out of the gated tier, and put the question in front of whoever owns the policy — support lead, risk, legal, product. Then write the answer into notes with a name and a date.

These are the highest-value cases in the suite. Half of what your agent gets wrong is a question nobody ever answered, and the eval directory is where those questions become visible instead of getting improvised at runtime.
A golden dataset is supposed to change — deliberately
Expectations do get updated: the policy changed, the old case was wrong, the agent genuinely improved. What makes it legitimate is that the edit is reviewable — same PR as the behaviour change, reason and decider recorded, visible in git log forever.

The reviewer question for any PR that touches expect: does this change what we consider correct, or does it change what we admit we do?
You just converted your eval suite into a screenshot of current behaviour
Pasting the observed output into expect produces a permanently green suite that can never detect a regression again, because it now defines correct as whatever the agent last did. Every future bug will be encoded as an expectation the day it ships.

This is the trap the last section of this note is about, and it never arrives labelled. It arrives at 17:40 on a Thursday, as one small edit that unblocks a release.

How it grows: one rule, attached to something you already do

Do not schedule eval work. Scheduled eval work gets bumped by the roadmap in three sprints. Attach the growth to rituals that already survive:

Every incident becomes a case, in the incident, before the retro. Add a line to your incident template — link to the eval case reproducing this — and make it a required field for closure. The case is written from the trace of the actual failure while the details are still fresh and nobody is yet arguing about whether it was really that bad. Ten incidents later you have thirty cases, all of them earned.

Every complaint that survives triage becomes a case. Same mechanism, lower stakes: whoever closes the ticket drops the input and the correct outcome into the directory. It takes four minutes and it is the cheapest institutional memory you will ever buy.

Every UNDECIDED case gets an owner and a date. They are open product questions; treat them like open tickets or they quietly become permanent.

Prune as well as grow. A case that has passed every run for six months and covers behaviour three other cases also cover is costing you runtime and telling you nothing new — retire it into the nightly tier. Thirty to sixty well-chosen cases holds a suite for a long time. If you find yourself at three hundred, that is when a real platform earns its keep and not before.

The trap: optimising the metric instead of the product

Six weeks in, the suite reports 19/20 and support is filing the same volume of complaints as before. Nothing was faked. What happened is the ordinary drift of an easier target displacing a harder one, and it shows up as a set of individually reasonable moves:

The judge rubric got tuned until it stopped disagreeing with the agent — a day of work on the measuring instrument to fix a problem in the thing being measured.
Two hard cases were moved to the nightly tier because they were "not representative", and the nightly tier is a report nobody reads.
An expectation got edited to match observed output at 17:40 on a Thursday.
The prompt grew a paragraph that helps precisely on the phrasings in the case files — twenty cases is a small enough target to overfit by hand.

Three counterweights, none of them expensive. Keep a blind holdout: ten cases nobody is allowed to look at while iterating, run monthly. When the gated score climbs and the holdout does not, you are polishing the metric. Keep one number the suite cannot influence — complaint volume, escalation rate, cost per successful run, measured on production traffic. Review the diff on expect fields like you review a database migration, because that is what it is: a change to your definition of correct.

And keep the framing straight, because it is the whole discipline in one line. The suite is not the product. It is a witness. The moment you start improving the witness instead of the thing it testifies about, you have a green pipeline and an agent your customers still do not trust — and you have lost the only thing the week was for.

Friday, 16:00, week one: twenty cases, forty deterministic checks, one trajectory check, one uncalibrated judge in report-only mode, and a gate at 16/20 with two named bugs. That is a real eval suite. It is not impressive, and it will catch the next regression before your customers do.