The statistical core: gating a system that answers differently

n = 1: what the gate actually tells you

Worked numbers, illustrative — the shape is what matters, not the decimals.

Your gate has 20 cases, one run each, and blocks below a 90% pass rate. So it blocks when 3 or more cases fail. Suppose the build is healthy: each case genuinely succeeds 95% of the time.

20 cases x 1 run, block below 90%  (i.e. block at >= 3 failures)

healthy build,   true per-run failure 5%   ->  gate blocks  7.6% of the time
regressed build, true per-run failure 15%  ->  gate blocks 59.5% of the time

Read both numbers. The healthy build is blocked roughly one release in thirteen for no reason at all — pure sampling. And a build that really did degrade from 95% to 85% sails through two times in five. That is the gate: it cries wolf once a fortnight and misses 40% of real 10-point regressions.

Then watch what the false alarms do to the humans. After the third spurious block, the team learns that this check is unreliable, and the override becomes the normal path — which is how a noisy gate converts into no gate at all.

n = 5: the same threshold, 100 observations

Keep the 20 cases and the 90% threshold. Run each case five times.

20 cases x 5 runs = 100 observations, block below 90%

healthy build,   true per-run failure 5%   ->  gate blocks  1.2% of the time
regressed build, true per-run failure 15%  ->  gate blocks 90.1% of the time

False alarms fall from 7.6% to about 1%. Detection of the same 10-point regression rises from 60% to 90%. Nothing about the agent changed — only the number of samples the gate looked at before deciding.

Two honest caveats, because this arithmetic is a simplification. First, it treats every run as independent; five runs of the same case are correlated, so real-world gains are smaller than the table implies — which is why you add cases as well as runs. Second, five runs costs five times the tokens and roughly five times the wall clock, and lesson five is about the fact that a gate too slow to run is worse than a gate that is slightly noisy. The design question is not “how do I make it perfect” but “what is the smallest regression I need to catch, and does my gate have the resolution to see it?”

Picking a threshold you can defend

Four steps, in this order. Skipping step 1 is why most agent thresholds are round numbers somebody liked.

1. Get the noise floor from the suite.
   evals - regression suites in CI gives the procedure; the gate
   just consumes the number. Anything smaller is not a signal.

2. Decide the smallest regression worth blocking.
   "A 2-point drop in the fast tier" is a decision, not a fact —
   it depends on what the agent does and who it does it to.

3. Check the resolution.
   On 20 cases x 1 run, ONE case is 5 points. A 2-point threshold
   cannot exist on that suite. Raise N, add cases, or widen the threshold.

4. Express it against a stored baseline.
   pass_rate >= baseline - 2.0pt, where baseline names the dataset
   version, model snapshot, prompt hash and run count that produced it.

And keep the three threshold shapes separate, because they answer different questions — the next block spells them out.

Publish the gate’s own reliability, not just the agent’s

Retry policy and the infrastructure-versus-subject flake distinction belong to the suite, and evals · regression suites in CI sets them out: retries are recorded and counted, never silent. The gate inherits those rules; what the gate owes on top of them is a number about itself.

So measure the gate as a system and publish it: runs per week, false-alarm rate per check, median duration, override rate per check. Those four numbers are what let you defend a blocking verdict to the engineer it just stopped, and they are the only evidence that distinguishes "this check is strict" from "this check is noisy". A gate whose false-alarm rate nobody has measured is a gate whose authority nobody can defend — and, per lesson one, a check with a high measured bypass rate is a design finding rather than a discipline problem.

What the gate needs from a baseline — and what it should refuse

Baseline hygiene is the prerequisite module's subject: evals · regression suites in CI covers what a baseline record contains and why a rate is meaningless without the ruler that produced it. Two rules are the gate's own, and both are refusals.

Refuse to run without a stored baseline. A threshold expressed against a remembered number is not a threshold. Refuse to compare across rulers. If the dataset version, the model snapshot or the run count differs between baseline and candidate, the gate does not emit a verdict — it demands the re-run, because the alternative is blocking a build that never regressed and then weakening the threshold to make the block go away.

Does the gate get a single number or a per-case diff?

Both, and the diff is the part that changes decisions. Post the per-case diff where the reviewer already is, and let the gate block on a must-pass flip even when the aggregate is within tolerance.

That combination — aggregate green, one case red, blocked — is the single most valuable behaviour in a statistical gate. Without it, the reviewer reads one green summary line, concludes the release is fine, and ships a security regression hidden underneath an average that was designed to absorb exactly that much movement.

How do you gate a judge-scored tier at all?

Carefully, and usually as warn rather than block. An LLM judge has its own variance and its own drift, so a 3-point move in a judge score is not comparable to a 3-point move in a deterministic assertion until you have measured the judge’s spread.

If you do want a judge tier to block, you need two extra things first: a calibration set with human labels, and a check that the judge still agrees with those labels this week (judge drift is a check in its own right — lesson three). Until those exist, a blocking judge threshold is a number with unknown error bars deciding whether you ship.