What triggers a run (a model upgrade is a code change)

1 · Prompt change — the system prompt, few-shot examples, output instructions

The most common trigger and the least local. Prompt edits interact: the sentence that fixes your refund case can flatten your escalation case, because both are decided by the same weights reading the same instructions.

Run: the fast tiers on the whole dataset, not just the cases you were fixing. If the edit touches wording, tone or refusal behaviour, add the judge tier before merge — this is one of the few times a judge belongs on the commit path.

2 · Tool change — and it is three different changes wearing one name

Implementation (the body of the function): ordinary unit tests catch most of it, and the model never sees the diff. Schema (argument names, types, required-ness): you changed the contract the model must satisfy, so every case that calls the tool is in scope. Description or docstring: this text sits in the context window on every single turn, so editing it is editing the prompt — treat it exactly like trigger 1.

Run: unit tests always; the tool-call tier for every case touching that tool; the full fast tier if you touched the description.

3 · Model version change — the code change with a diff you cannot read

A new model version rewrites the decision-making core of your system. Nothing in your repository moved, so nothing in your habits fires — which is why this class deserves the loudest alarm in your pipeline. Regressions from a model swap are famously uncorrelated with the improvement claims: a version that is better on public benchmarks can be worse at obeying your specific tool-selection rules, or newly chatty in a field your parser truncates.

Run: the full suite on both versions with everything else pinned, then diff per case rather than comparing headline rates. Budget for a staged rollout afterwards.

4 · Framework or SDK bump — the harness decides what the model sees

Your agent framework assembles the context, serialises tool schemas, applies retry logic, truncates history and parses tool calls. A minor version can change any of those without a line in its changelog that reads “changes model behaviour”.

Run: the fast tiers on the dependency-bump pull request. Automated dependency PRs that only run unit tests are the most common way a behaviour regression walks straight into main.

5 · Retrieval content and index changes — an input nobody commits

For a retrieval-backed agent, the corpus is an input to every run. Re-chunking, a new embedding model, a rebuilt index, or a product manager rewording one policy article all change what the model reads.

Run: on index rebuilds, the grounded slice of the suite. And schedule a periodic run even with no trigger at all, precisely because content edits happen outside your pipeline.

6 · Inference config — temperature, max tokens, truncation, timeouts

Config is code with worse ergonomics. Raising temperature widens the action distribution; lowering max tokens truncates answers your assertions expect; a shorter tool timeout converts slow successes into failures the model must now recover from.

Run: the fast tiers, and specifically re-measure variance — config changes are the ones most likely to move your flake budget rather than your pass rate.

7 · The suite itself — the change that invalidates your history

Adding ten hard cases lowers the pass rate mechanically. Rewording a judge rubric moves every judge score. Both are legitimate, and both break comparability.

Run: version the dataset and the graders alongside the code, record the version in every report, and never compare a rate across dataset versions without saying so. When you add cases, re-baseline: run the new suite against the previous build so the number you compare to was produced by the same ruler.

A change just landed. What should it trigger?

Interactive decision tree — outcomes:

• Unit tests plus the fast tier — merge on green

  Nothing the model can perceive moved, so the eval suite has no new information to give you. Run the deterministic base anyway (it is nearly free and it catches the case where you were wrong about “behaviour-preserving”), and let the nightly full run be the backstop.

• Fast tiers on everything, plus every case touching that tool

  Run the cheap tiers across the whole dataset — scoped changes leak more often than anyone expects — and treat the cases that exercise the changed tool as the ones that must not regress. Nightly picks up the judge tier. Total cost: minutes.

• Full suite before merge, judge tier included

  A global change invalidates every score you have. Run the whole dataset with N runs per case, include judge grading, and compare against a baseline produced by the same dataset version. Post the per-case diff on the pull request so the reviewer sees which cases flipped, not just the headline rate.

• The highest-risk class you own — treat it as a project, not a bump

  Run the full suite on old and new versions with prompt, tools, dataset and config pinned identically, then diff per case: expect a handful of regressions that have nothing to do with the advertised improvements, and read their traces individually. Aggregate parity is not evidence of per-case parity. Then stage the rollout behind a canary with a kill switch and watch online metrics, because the offline suite cannot represent your whole traffic distribution.