Designing your own gate — and its limits

Gate depth by change class — spend the deep tiers where the change earns them

1. Tier 3 — security review, shadow, staged rollout — days · rare · a human decision is part of the gate

   Reserved for changes that alter the security posture or the decision-making core: a new write-capable tool, a permission widening, a third trifecta leg, a model version change, a new MCP server.

   Everything from tier 2, plus: security review with the permission diff attached, shadow evaluation against mirrored production traffic with side effects disabled, then a canary with a kill switch and auto-halt thresholds, then feature-flagged cohorts before full traffic — the ladder in agentops · rollout and kill-switches, whose rungs the gate does not get to skip. For a model change, add the per-case diff between old and new snapshots — aggregate parity is not per-case parity.

2. Tier 2 — full suite × N runs, judges, cost report — tens of minutes · at the release candidate

   For global changes that invalidate every number you hold: a system-prompt edit, a framework bump, an index rebuild, a parameter change.

   Full golden dataset at your chosen N, judge tier included (as warnings), cost and latency report, per-case diff against a stored baseline of the same dataset version. Plus the two checks that need real infrastructure rather than a manifest: the egress-allowlist and sandbox probes, fired against the deployed candidate environment — attempt a connection off the list, attempt a write outside the workspace, and assert both fail. Runs on the candidate artifact with the versions it will actually ship with — not on main.

3. Tier 1 — fast eval slice + the must-pass fixtures — minutes · every push that touches the tuple

   Tool-call correctness and output assertions on a fast slice at 3–5 runs per case, plus the full must-pass set — injection and forbidden-action fixtures included, at the same N. Anything that runs the agent is minutes rather than seconds: the gate spec in lesson one budgets the 47-fixture set at 180 s for that reason.

   Enough to catch an obvious break, deliberately not enough to measure a two-point regression — that is tier 2’s job, and pretending otherwise is how thresholds get set inside the noise floor.

4. Tier 0 — manifest checks: contracts, diffs, pinning — seconds · every push, no exceptions

   Everything decidable from the resolved manifest alone: contracts parse, the inventory, permission and trifecta-leg deltas are computed and attached, nothing resolves to a floating alias or a mutable tag.

   No model calls and no deployed environment — which is exactly why this tier stays under a minute. The fixtures do not live here: they run the agent, so they cost N runs and belong in tier 1. Nor do the egress and sandbox probes, which have to be fired against deployed infrastructure and land at the release candidate with tier 2. This is the only tier that will still be running unmodified in two years, and it is the tier that makes the human review at tier 3 cheap.

Every incident becomes a check — and the check is written before the postmortem closes

This is how a gate acquires teeth, and it is the only reliable mechanism anyone has found. The agent leaked an internal note into a customer reply: the exact transcript becomes a fixture with a negative assertion. The tool timed out and the agent invented an order status: the timeout becomes an injected-error case. A prompt edit flattened escalation behaviour: the escalation case joins the must-pass set.

Two rules make it stick. Write the check while the incident is open, because the specific inputs are only available then and the motivation evaporates by Thursday. And make the check fail first on the unfixed build — a regression test you never saw red is a regression test you cannot trust.

The same discipline runs through evals · eval fundamentals as the eval lifecycle. The gate is where those cases acquire authority: a case in the dataset informs you, a case in the must-pass set stops you.

Gate runtime is a design constraint, not a discipline problem

A 40-minute gate gets bypassed. Not because your engineers are undisciplined, but because a 40-minute wait between “I think this is right” and “I know” is longer than human working memory, so people batch changes, skip the gate for “trivial” edits, and lobby for a skip label. Then the batching makes every failure harder to attribute, which makes the gate less useful, which makes bypassing it more attractive.

What makes this harder for an agent gate than for a unit-test suite is the arithmetic from lesson two. Your wall clock is cases × N × model latency and your bill is cases × N × tokens, so the same knob you turn to buy resolution — raising N from 1 to 5 to drop false alarms from 7.6% to about 1% — multiplies both, and provider rate limits can flatten the parallelism you were counting on to hide it. Resolution and speed genuinely trade against each other here, which is why the expensive tiers have to move off the critical path rather than be made faster: there is no optimisation that makes five samples cost one.

Design to a budget instead. Tier 0 under a minute. Tier 1 under ten. Tier 2 off the critical path — nightly and at the candidate. If the blocking path exceeds your budget, the fix is to move checks to a later tier or make them cheaper, never to add a skip label. Publish the gate’s median duration next to its override rate: those two numbers together tell you whether the gate is alive.

Gate theatre: the checks that always pass

A check that has never failed is one of three things, and it is worth knowing which. It might be a genuine invariant enforced somewhere else (fine — but then it is documentation, not a control). It might be miscalibrated so far in the safe direction that nothing could trip it (a threshold of 50% on a suite that runs at 92%). Or it might not be testing what its name says (an assertion on a field that is always present, a fixture whose payload was paraphrased into harmlessness).

Audit for it mechanically: record each check’s fire rate and review the zeros quarterly. Then run the honest experiment — deliberately introduce the defect the check claims to catch and confirm it goes red. A gate nobody has ever seen fail is indistinguishable from a gate that does nothing, and the two feel exactly the same right up to the incident.

What the gate cannot do, stated plainly

A gate catches known-bad. Fixtures replay attacks you have seen. Thresholds detect regressions against behaviours you encoded. Diffs flag deltas from a state you approved. All of it is retrospective by construction.

What bounds the unknown is containment, and it is a different kind of engineering with its own catalogue: security · guardrails catalog enumerates the layers and the order to add them in, so this module does not repeat the list. The property to carry across is that every one of them bounds damage rather than detecting badness, which is why none of them can be a gate check and why the gate cannot stand in for them.

The relationship is not a trade-off. A gate with no containment behind it is a promise that you thought of everything; containment with no gate means every known regression reaches production and gets caught by the blast shield. You need both, and the gate’s real value is that it keeps the containment layer’s workload small enough to reason about.

One change is ready. How deep does the gate go?

Interactive decision tree — outcomes:

• Tier 3 — block, route to security, then shadow and canary

  The gate stops here on purpose. Attach the inventory diff, the permission diff and the recomputed trifecta leg count, and hand a human a delta rather than a system. If review approves, run the full suite, then shadow the candidate against mirrored traffic with side effects disabled, then canary behind a flag with a kill switch. The eval numbers are necessary and not sufficient: no pass rate tells you whether this capability should exist.

• Tier 3 — treat the model change as a migration, not a bump

  Run the full suite on both snapshots with prompt, tools, dataset, parameters and framework pinned identically, then diff per case and read the traces of the flips — expect a handful of regressions unrelated to the advertised improvements. Re-baseline judge scores, because a threshold calibrated against the old model is not a comparison. Then stage the rollout with auto-halt metrics. Your gate should refuse to score a model change against the old baseline at all.

• Tier 2 — full suite × N runs at the candidate, judges as warnings

  A global change invalidates every number you hold, so run the whole dataset at your chosen N against a baseline of the same dataset version, post the per-case diff on the pull request, and let must-pass failures block even when the aggregate is inside tolerance. Judge tier attaches as advisory with an owner. Tier 0 still runs — it is the cheapest thing in the pipeline and it catches the case where you were wrong about “global”.

• Tier 1 — fast slice plus the full must-pass set

  Anything the model reads is a prompt change, including a tool description and a tool’s error text. Run the fast tier across a slice at 3–5 runs per case, run every must-pass case, and let the nightly full run be the backstop. Minutes, not tens of minutes — which is what makes it survivable on every push.

• Tier 0 — contracts, diffs, pinning. Merge on green

  Nothing the model can perceive moved, so the eval suite has no new information to offer. Run the manifest tier anyway: it is nearly free, needs no model calls, and it catches the case where “behaviour-preserving” was an assumption rather than a measurement. If the tool’s bytes, wording, ordering or error text changed, you were in tier 1 and did not know it.