The gate as an artifact

Lesson 1 of 5 in Release Validation Gates: What Stands Between a Change and Production.

You have a regression suite that runs in CI. You have guardrails in the runtime. You have a canary plan. What you may not have is the thing that connects them: one artifact that decides whether this change is allowed to move.

A validation gate is not a culture, a checklist in a wiki, or a habit of asking Priya to look at the eval report. It is a data structure:

an ordered list of checks, each with a verdict class (block, warn, record), a runtime budget, an owner, and a rule for what evidence it emits.

The test of whether you have one is blunt: can you print it? If the answer is “well, we run the evals and then usually someone from security looks at new tools,” you have a set of practices whose coverage nobody can state and whose failures nobody can count.

Write the gate down and three things become possible that were not possible before. You can diff it — a check added after an incident is visible as a line. You can measure it — how long it takes, how often each check fires, how often each is overridden. And you can argue about it in advance, at design time, rather than under deadline pressure with a release manager asking whether a 2-point drop counts — the same argument agentops · rollout and kill-switches makes about writing the promotion contract before the canary starts instead of during it.

The gate is also the thing that makes human review affordable. This is worth being explicit about, because gates get sold as a way to remove people: they are not. A gate cannot judge whether a new cancel_subscription tool is a sensible product decision. What it can do is guarantee that a human who must judge that arrives with the tool diff, the permission delta, and the eval numbers already on screen — so the review is ten focused minutes instead of an hour of archaeology, and so the reviews that do not need a human never reach one. The autonomy-and-oversight framing behind this lives in architectures · human-in-the-loop; here it is a design constraint: every blocking check should either be machine-decidable or route to a named human with the evidence attached.

A gate, written down (illustrative shape — not a product schema) (schema)

gate: refund-triage / promote-to-prod
budget_minutes: 12          # wall clock for the whole gate

checks:
  - id: tool-contract
    verdict: block
    owner: agent-platform
    budget_s: 20
    evidence: schema-report.json

  - id: tool-inventory-diff
    verdict: block-on-add    # new or widened tool blocks; removals warn
    owner: agent-platform
    budget_s: 10
    evidence: inventory-diff.md

  - id: permission-diff
    verdict: block
    owner: security
    budget_s: 10
    evidence: iam-diff.md, trifecta-legs.json

  - id: version-tuple-pinned
    verdict: block
    owner: agent-platform
    budget_s: 5
    evidence: manifest-resolved.yaml

  - id: injection-fixtures        # 47 payloads that once worked
    verdict: block               # must-pass: zero tolerance, any run
    owner: security
    budget_s: 180
    evidence: forbidden-action-report.json

  - id: eval-fast-tier            # 40 cases x 5 runs, pinned model
    verdict: block
    threshold: pass_rate >= baseline - 2.0pt   # aggregate vs baseline
    floor: pass_rate >= 0.85                   # absolute floor, history-free
    flake_budget: max_flaky_cases: 5           # stability, not rate
    owner: agent-quality
    budget_s: 420
    evidence: eval-report.json

  - id: judge-tier                # faithfulness + tone rubric
    verdict: warn
    owner: agent-quality
    budget_s: 900                 # runs off the critical path
    evidence: judge-report.json

  - id: cost-latency-budget
    verdict: warn
    owner: agent-platform
    budget_s: 0                   # computed from the eval run
    evidence: cost-report.json

overrides:
  allowed_by: [release-manager, security-oncall]
  requires: [reason, linked-ticket, expiry<=14d]
  # forbidden_for must list every no-override row from lesson three,
  # or the artifact and the prose have already drifted apart
  forbidden_for: [injection-fixtures, permission-diff, version-tuple-pinned]
  logged_to: release-audit-log     # append-only

Two lines carry most of the weight. verdict is declared per check, in advance — so nobody negotiates blocking-ness while a release is waiting. And forbidden_for names the checks that no override can clear: the security-class checks where "usually refuses" is not a property worth having, plus tuple pinning, because an unpinned release means every other check in the list was run on something else. Keep that line synchronised with the no-override rows in lesson three by hand if you have to; a gate whose printed exemplar permits what its own text forbids is the folklore this module is arguing against. Notice also that every check names an owner and an evidence artifact. A check with no owner rots; a check with no evidence produces "gate failed" and gets retried until green.

Three verdict classes. Every check declares exactly one, before it is ever run in anger.
Verdict	What it means mechanically	Use it when	How it fails
Block	The pipeline stops. The artifact does not move. Clearing it needs a fix or a logged override.	Deterministic checks, security invariants, and statistical thresholds you have measured enough to defend.	Blocking on a noisy signal. Engineers learn the check is unreliable, and the override becomes the normal path.
Warn (advisory)	The pipeline continues. The finding is attached to the release and assigned to the check’s owner with a deadline.	Expensive or variance-carrying graders (judge scores), cost and latency trends, new findings whose baseline you do not trust yet.	Warnings with no owner and no deadline. After a month the release page shows 40 warnings and nobody reads any of them.
Record	No verdict at all. The value is written to the release record so it can be compared later.	Baseline-building: a new check’s first weeks, token spend per run, turn-count distributions, tool-call frequency.	Recording forever. A metric nobody ever promotes to warn or block is telemetry you pay for and never use.

A change travelling through the gate to a canary

Change lands (prompt · tool · model · config)
The trigger classification from evals · regression suites in CI decides which checks this change needs. A model version bump is a change even though no file moved.
Resolve version tuple → immutable manifest
Prompt hash, tool schemas, pinned model snapshot, parameters, framework build. Everything downstream validates this artifact, not "main".
Deterministic checks: contracts · diffs · pinning
Seconds. No model calls. Tool schemas parse, the tool inventory diff is reviewed, permissions are unchanged or approved, nothing resolves to a floating alias.
Permission diff adds a trifecta leg?
Counted mechanically from the manifest: private data, untrusted content, external communication. Going from two legs to three is a security-review trigger, not a merge comment.
Route to security review with the diff attached
A human decision the gate cannot make — but arriving pre-packaged: which tool, which scope, which leg, which alternatives were considered.
Eval tiers: fixtures · fast suite × N runs · judges
Injection fixtures and forbidden-action assertions are must-pass. The fast suite is compared against the recorded baseline. Judge scores attach as warnings.
Thresholds held? (rate · must-pass · flake budget)
Three questions, not one: did the aggregate move beyond tolerance, did any must-pass case fail on any run, did the flake count grow?
Blocked · evidence + per-case diff attached
A useful block names the cases that flipped, links their traces, and says which check owns the verdict. "Gate failed" with a log dump gets retried, not read.
Promote to canary (small share, auto-halt)
The gate has done all it can offline. Online metrics and a kill switch take over — agentops · rollout and kill-switches.
Feature-flagged cohorts
The next rung of the rollout ladder in agentops · rollout and kill-switches: named cohorts behind a flag, with promotion to full traffic owned by a human rather than a threshold. The gate does not skip rungs on the pipeline's behalf.
Full traffic

Who may override, and what must they write down?

Overrides are legitimate. A gate with no escape hatch gets disabled entirely the first time it blocks an outage fix, so design the hatch instead of pretending you will not need it.

Name the roles in the gate definition — release manager, security on-call — and require four fields: which check, why (a sentence about the specific finding, not “urgent”), a linked ticket for the follow-up, and an expiry. The expiry is the part teams skip and the part that matters: an override with no end date is a permanent silent change to your gate, made by one person, at speed.

Which checks may never be overridden?

Pick a small set and mean it. The usual members: injection and forbidden-action fixtures, permission and trifecta-leg diffs, and version-tuple pinning. These are the checks whose failure means the release is not the thing you tested, or that a security property you claimed is no longer true.

Enforce it in the tooling, not in a policy document. If the override command can technically clear a must-pass check, it will, at 2 a.m., with a good reason that turns out to be wrong.

Why does an unlogged override mean you do not have a gate?

Because a gate is only as strong as its measured bypass rate. If overrides are not recorded, you cannot answer the two questions that tell you whether the gate is real: how often is each check overridden, and which check is overridden most?

A check overridden in a third of releases is not protecting anything — it is either miscalibrated (too noisy), misplaced (blocking when it should warn), or testing something the team disagrees with. All three are fixable. None are visible without a log. Append-only, one line per override, with the four fields above: that is the minimum honest implementation.

What does the gate owe the human it escalates to?

Three things, and they are cheap to produce because the gate already computed them: the diff (what changed in the tuple), the delta (what moved in the numbers, per case, against a named baseline), and the reason this needs a person (which check routed it, and what decision is being asked for).

Compare that with the common alternative — a Slack message saying “can you take a look at the evals before we ship?” — and you can see why review quality is mostly a gate-design problem. An approval gate where the reviewer has to reconstruct the context is the one that decays into a reflex click.

Key terms: promotion gate, quality gate, approval gate, human-in-the-loop, blast radius, trace

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.