The eval lifecycle: every incident becomes a case

Lesson 6 of 6 in Eval Fundamentals: You Cannot Improve What You Cannot Measure.

An eval suite is not a project you finish. It is a ratchet: every failure the world shows you gets converted into a case, and the suite only ever gets harder. That is the whole mechanism by which an agent gets better over months rather than oscillating forever.

The loop is short enough to memorise. Something goes wrong in production. You have the trace, because you recorded it. You reduce that trace to a case: the input, the context, the expected outcome, the forbidden actions. You add it to the suite and confirm it fails — an incident case that passes on the first try means you reproduced the wrong thing. Then you fix, and the suite tells you both that the case now passes and that nothing else broke. The case stays forever.

Incident to permanent test case

Production failure observed
A customer complaint, an alert, a reviewer override, or someone reading traces and going “huh”.
Pull the trace of the bad run
This step is only possible if you were already recording. An unrecorded incident can be apologised for but not learned from.
Reduce to a case: input, context, expected outcome, forbidden actions
Strip the identifiers, keep the shape. Decide what should have happened — this is where a human supplies ground truth.
Case reproduces the failure?
Run it several times. A 3/10 reproduction is still a valid red case; a 10/10 pass means your case is not the incident.
Sharpen the case — more of the real context
Usually the missing ingredient is context: the long history, the specific retrieved document, the tool output that was truncated.
Fix — prompt, tool, context, or runtime control
If the incident was an invariant violation, the fix belongs in the runtime; the eval case is the sentinel, not the guardrail.
Full suite green, no regressions
Case joins the suite permanently
It now runs on every change, forever. This is the ratchet: the agent cannot silently regress into a failure you have already paid for.

Where the suite runs determines whether the discipline survives contact with a deadline. A suite you run by hand when you remember is a suite that stops existing in week three. The endpoint is a regression suite in CI: a fast slice on every pull request, the full suite nightly, and a hard gate before any change to the model, the prompt, or the tool set reaches production. The mechanics of doing that without burning your token budget — slicing, caching, parallelism, flake policy — are the next module’s job.

One more habit worth stealing from mature teams: run the suite against production traffic continuously, not only against the frozen dataset. Sampled online scoring catches the drift that a static dataset cannot see, because the world changes the inputs even when you change nothing. Your golden dataset measures regressions in your work; online scoring measures regressions in the world.

Where evals run, and what each placement is for
Placement	What runs	Trigger	What it protects
Local dev loop	A slice of 5–20 cases, 3 runs each, deterministic graders only	Every prompt edit, by hand or on save	Your own time — fast feedback on whether the last change helped
Pull-request gate	Core slice plus the entire safety slice, judges included	Every PR touching prompts, tools, or agent code	The main branch. A PR that drops the safety slice does not merge
Nightly / pre-release full suite	Every case, higher run counts, cost and latency recorded	Schedule, and before any release	The release decision, and the autonomy level it licenses
Model or dependency upgrade	Full suite and judge re-calibration against human labels	Any change of model, model version, or judge model	Comparability. A new model silently changes both the agent and the grader
Online scoring in production	Sampled live runs scored continuously; managed services offer this on traces	Continuous, on a sample of real traffic	Against drift in the world — new intents, new phrasing, changed upstream data

Tooling — a September 2026 snapshot, not a recommendation

You do not need a platform to start; a directory of case files and a script that records rates is a real eval suite. When you outgrow it, the landscape splits roughly into open-source-first tools you can self-host — Langfuse (MIT core, tracing plus code and LLM-as-judge evaluators, datasets) and Arize Phoenix (OpenTelemetry-based, self-hostable, licensed under the Elastic License 2.0 rather than a permissive OSI licence) — and commercial platforms such as LangSmith and Braintrust. Cloud providers ship their own layer: Amazon Bedrock AgentCore Evaluations scores agent interactions from OpenTelemetry traces using LLM-as-a-judge by default, in on-demand, online-sampled and batch modes, and Microsoft Foundry offers built-in agent evaluators. Versions and feature gating move monthly — check the current docs, and note that the OpenTelemetry GenAI semantic conventions your traces depend on are still marked Development, not stable.

Interactive flashcard deck.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.