The mechanics: pinning, N runs, thresholds, diffs

A pipeline with eval gates

1. Push / PR opened

   The trigger classification from lesson two decides which tiers this change needs.

2. Lint · types · tool unit tests

   No model calls, no tokens, seconds of wall clock. Everything that can be caught here must be caught here.

3. Fast tier: schema · tool-calls · assertions

   Pinned model version, pinned framework, pinned dataset version, temperature fixed. A slice of the dataset, 1–3 runs per case, minutes not tens of minutes.

4. Within flake budget of baseline?

   Compare against the recorded baseline for this dataset version — never against a remembered number. Must-pass safety cases have no budget at all.

5. Blocked · report diff + failing traces attached

   A useful failure names the cases that flipped and links their traces. “Eval job failed” with a log dump gets ignored.

6. Merge to main
7. Nightly: full suite × N runs · judges · cost

   Where statistical honesty lives: enough runs per case to separate signal from sampling noise, plus the expensive graders.

8. File regression · add case to must-pass set

   A nightly regression becomes a tracked issue and, once diagnosed, a permanent case. This is how the suite grows teeth over time.

9. Release candidate ≥ release baseline?

   Run against the exact artefact you intend to ship, with the versions it will ship with.

10. Canary + shadow eval on live traffic

    The offline suite has done all it can. Real traffic is the next instrument — lesson five.

11. Promote to full traffic

Pin the run

Every variable you do not pin becomes an explanation for tomorrow’s mystery. Record the whole set in the report, not just in the config file, so an old number can be interpreted a year later.

# eval-suite.yaml  (illustrative — shape, not a real product schema)
model:        lm-large-2026-06        # a snapshot id, never a floating alias
temperature:  0
max_tokens:   1024
framework:    agent-sdk==2.4.1        # the harness assembles the context
prompt_ref:   prompts/triage@v17      # versioned like code
tools_ref:    tools/manifest@v9       # names + schemas + descriptions
dataset:      golden/triage@v6        # rates are only comparable within a version
judge:
  model:      lm-large-2026-06
  rubric_ref: rubrics/faithfulness@v3 # the judge is a dependency too
runs_per_case: 3
seed:          4211                   # best-effort where supported; see the caveat below

Score the run

Three kinds of threshold, and they answer different questions.

thresholds:
  overall_pass_rate:  ">= baseline - 2.0pt"   # regression tolerance, not an absolute floor
  must_pass:                                  # zero tolerance, any run, any version
    - injection/*                              # forbidden-tool assertions
    - pii/*                                    # no personal data in outputs
    - refund_cap/*                             # the invariant the runtime enforces
  flake_budget:
    max_flaky_cases: 5        # cases that pass on some runs and fail on others
    action: quarantine        # reported separately; never silently retried

Regression tolerance protects you from noise. Must-pass cases are invariants: one failure in one run blocks, because “usually refuses to exfiltrate” is not a security property. Flake budget makes instability visible instead of letting it hide inside an averaged rate.

Diff the run

Aggregate rates hide the interesting part. Two builds at 86% can disagree on twenty cases. Post the per-case diff where the reviewer already is — the pull request.

eval report · triage@v6 · 50 cases × 3 runs
  pass rate   84.7%  (baseline 86.0%, -1.3pt)   within tolerance
  must-pass   40/40  OK
  flaky       3 cases (budget 5)
  regressed → case 018 refund-over-cap      3/3 → 0/3   forbidden tool called
              case 041 multi-item-return     3/3 → 1/3   missing order id in reply
  improved   → case 007 duplicate-ticket     0/3 → 3/3
              case 022 policy-quote          1/3 → 3/3

That report blocks the merge on case 018 despite the aggregate being within tolerance — a must-pass case flipped. Without the diff, the reviewer sees “-1.3pt, fine” and ships a security regression.

How many runs per case? (and why one is a lie)

One run per case gives you a verdict on a coin flip you cannot see. Three to five runs is the usual CI compromise: enough to expose bimodal cases, cheap enough to keep the fast tier under a few minutes.

The principled version: pick N so that the suite’s run-to-run spread is smaller than the smallest effect you want to detect. If you want to catch a 3-point regression and repeated runs of the identical build swing 5 points, no threshold will save you — raise N, or add cases, or both.

Measure your noise floor before you trust any threshold

Run the identical build through the suite twice, ideally three times. The spread you see is your noise floor. Anything smaller than it is not a signal, no matter how much it looks like one, and any team that has not done this exercise is arguing about sampling error in review meetings.

Do the arithmetic too: on 50 cases at 3 runs each, one case flipping all three runs moves the rate by 0.67 points. On a 20-case suite, one case is a 5-point swing — so do not report a 20-case rate to one decimal place and do not build a 2-point threshold on top of it.

Flake budget: make instability a tracked number

A flaky case is one that passes on some runs and fails on others with everything pinned. Flakiness is information: it usually means the case sits on a genuine decision boundary — an ambiguous instruction, a borderline threshold, two defensible tool choices.

Count flaky cases explicitly, quarantine them out of the blocking rate, and track the count as its own metric. A build where the pass rate held at 86% but flaky cases went from 3 to 11 has regressed, and only the flake number will tell you.

Baseline hygiene — a number needs a ruler

A baseline is an artefact, not a memory: pass rate, per-case results, dataset version, model version, framework version, prompt version, judge version, run count, date. Store it, refresh it on every merge to main, and refuse comparisons across dataset versions unless you re-ran the old build on the new dataset.

Screenshots of a rate in a chat thread are the most common form of eval debt. Six weeks later nobody can say what produced the 91%.

Which failures block, and which just report?

Blocking on everything makes the suite the enemy; blocking on nothing makes it decoration. The workable split: must-pass cases and the deterministic tiers block; judge-scored aggregates, cost trends and flake counts report with an owner and a deadline.

Safety and compliance cases are the clean case for zero tolerance — a injection case that fails one run in twenty is a vulnerability with good luck, not a passing test.