Regression Suites in CI: Evals That Run On You

The discipline that turns a pile of eval cases into a gate nobody can walk around: a deterministic-first pyramid, per-commit and nightly tiers, pinned model versions, honest thresholds over N runs, and the offline-to-online bridge that catches what your golden dataset never will.

Content current as of 2026-09.

Lessons

  1. The eval pyramid: deterministic first, judges last
  2. What triggers a run (a model upgrade is a code change)
  3. The mechanics: pinning, N runs, thresholds, diffs
  4. Prompt and tool unit tests: the evals that need no model
  5. Offline to online: why green suites still ship regressions