The Regression Gate

Lesson 1 of 3 in Production Evals and Regression Testing.

Software teams settled a question decades ago: code does not ship without tests passing. This module applies the same discipline to LLM systems, and its rule fits in one sentence: no model swap, no prompt change, no retrieval or pipeline change ships without the Eval harness passing.

The rule earns its strictness because LLM changes are nonlocal. Editing a function changes that function; editing a System prompt changes, potentially, every output the system produces. A one-word tweak that fixes the complaint in front of you can quietly break behavior on inputs that share nothing with that complaint — different topic, different language, different format. There is no call graph to trace, so the only way to see a change’s blast radius is to measure it: run the Golden set before and after, grade both runs the same way, and compare.

The previous module built the machinery — the golden set sampled from real traffic, the Grader ladder from string checks up to LLM-as-judge. This module is about running that machinery as a gate: what a change has to survive between “works on my prompt” and production. Regression testing is the name, but the LLM version has two twists a software engineer won’t expect — the pass/fail line is statistical, and a human still reads the diffs.

Key terms: Regression testing, Eval harness, Golden set, Grader, Drift

A change moves through the regression gate

  1. Proposed change

    A model swap, a prompt edit, new sampling settings, a retrieval or pipeline change — anything that can alter what the model sees or says.

  2. Run the full harness

    The whole golden set, graded by the same grader ladder as the baseline run. Everything held constant except the change under test.

  3. Compare deltas to the noise band

    Rerunning the unchanged baseline several times measures how much scores wobble on their own. That spread — per metric — is the ruler. A delta inside it is noise, not signal.

  4. Signal-level regression on a gated metric?

    Each metric has its own threshold: hard floors for must-not-regress metrics (format validity, refusal behavior), tolerated dips for trade-off metrics (verbosity, style).

  5. Human reviews per-item flips

    Every case that flipped pass→fail or fail→pass gets eyes on it. Flips are usually few, so this is minutes of work — and it is where changes reveal what they actually do.

  6. Flips acceptable?

    Losing three edge cases to gain ten common ones may be a fine trade — or a catastrophe, if the three are your largest customer’s workflow. A human owns this judgment.

  7. Ship — this run becomes the new baseline

    Store the scores and the per-item outputs. The next change diffs against this run, and lesson three’s migrations depend on these stored baselines.

  8. Blocked — revise and resubmit

    Fix the change, or bring evidence that a threshold should move. Overriding the gate by hand every week means the thresholds are wrong — renegotiate them explicitly.

The step software engineers trip on is the noise band. A unit test either passes or fails; an eval score wobbles. Run the identical harness on the identical system twice and the numbers can differ: Sampling at nonzero Temperature produces different outputs by design, many serving stacks do not guarantee bit-identical results even at temperature zero, and any LLM-as-judge grader adds verdict flakiness of its own on borderline cases. So the gate never asks “is the new score equal to the old one?” It asks “is the difference bigger than the differences we see when nothing changes?” Measuring that means rerunning the unchanged baseline a few times per metric and recording the spread — boring, cheap, and the foundation the whole gate stands on. Skip it and the gate fails in both directions: it blocks harmless changes on phantom regressions, and teams respond by learning to ignore it.

Thresholds are the second twist: they are per metric, not one blended score. A schema-validity metric guarding downstream parsers is a hard floor — any signal-level regression blocks. A verbosity or tone metric might tolerate a stated dip when task success improves. Collapsing everything into a single number recreates, inside your own team, exactly the Benchmark problem this domain keeps warning about: one number invites optimizing the number.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.