The agent-specific checks nobody else will tell you to run

Tool-contract check

Deterministic, no model, runs in milliseconds. Assert on the manifest the model will actually see.

FOR each tool in manifest:
  parses as JSON                          -> else FAIL
  schema is valid JSON Schema (draft X)   -> else FAIL
  name matches ^[a-z][a-z0-9_]{2,63}$     -> else FAIL
  description non-empty, >= 20 chars      -> else FAIL   # the model reads this
  every property has a type               -> else FAIL
  every enum non-empty                    -> else FAIL
  required[] names existing properties    -> else FAIL
  no additional undeclared params         -> else WARN
  write-capable tools declare a cap       -> else FAIL   # e.g. max_amount

manifest-level:
  tool names unique                       -> else FAIL   # renames break saved prompts
  total serialised size <= budget         -> else WARN   # context cost is real

Two assertions people leave out. Empty descriptions: a tool with a blank description still executes perfectly and is never chosen well, which surfaces as a mysterious quality drop. Missing enums and patterns: dropping enum: [read, write] from a mode argument does not break anything at build time — it widens what the model is allowed to ask for, and schema validation is the only thing that was narrowing it.

Inventory + permission diff

Both diffs run against the last released manifest and the last approved permission envelope, stored as artifacts.

tool inventory diff  refund-triage v7 -> v8
  + cancel_subscription        WRITE   scope: billing:subscriptions.*
  ~ search_kb                  args: +include_attachments (bool)
  ~ lookup_order               description changed (hash a11b -> c4d9)
  - legacy_refund              removed

permission diff
  + billing:CancelSubscription  on  arn:...:subscription/*
  ~ storage:GetObject           resource  tickets/*  ->  tickets/**, exports/**
  = egress allowlist            unchanged (3 destinations)

trifecta legs  (computed from tools + data sources + egress)
  private data          YES   lookup_order, storage:GetObject on tickets/*
  untrusted content     YES   search_kb reads customer attachments
  external comms        NO    -> was 2 legs
                        YES   cancel_subscription posts to partner API
  VERDICT: 2 -> 3 legs.  BLOCK. route: security-review

Three findings in one screen, and the last one is the one a scanner will never produce. Note what triggered it: not a vulnerability, not a misconfiguration — a useful feature that happens to complete an exfiltration path. That is the class of risk only a gate that understands agents can see.

Version-tuple pinning check

The cheapest high-value check in the whole gate. It asserts that the artifact is pinnable — that nothing in it can move after you measured it.

resolve manifest -> assert every element is immutable

model.id            matches  <provider>/<model>@YYYY-MM-DD   -> else FAIL
model.id            NOT matching  *-latest | *-preview | @alias -> else FAIL
parameters          present in manifest, not from env          -> else FAIL
prompt.rendered_sha present and matches build output           -> else FAIL
tools[].schema_sha  present for every tool                     -> else FAIL
framework.sdk       exact version, no ^ or ~                   -> else FAIL
framework.image     digest (sha256:...), not a tag             -> else FAIL
mcp_servers[]       pinned version + description hash          -> else FAIL

# and the check that catches the interesting case:
baseline.model.id == candidate.model.id  -> else REQUIRE model-upgrade path

That last line is why this check earns a block. When the model snapshot differs from the baseline’s, the gate must refuse to treat the comparison as a like-for-like regression run and demand the model-upgrade procedure instead: full suite on both versions, per-case diff, staged rollout. Otherwise a model bump is validated by a threshold that was calibrated against a different model — a comparison with no meaning at all.

Judge calibration: your grader is an instrument, and instruments drift

If a check’s verdict depends on an LLM judge, the judge is part of your gate and needs its own validation. Keep a frozen calibration set — 30 to 60 cases with human labels — and re-score it whenever the judge model, the judge prompt, or the rubric changes, plus on a schedule for the changes you did not make.

Report agreement with the human labels as a number, and treat a fall in agreement as invalidating the judge tier, not as invalidating the release. The failure mode this prevents is nasty and common: the judge score drops four points, the team spends a week hunting an agent regression, and the actual change was a judge model upgrade. Version the rubric and the judge model in the manifest alongside everything else — judge drift is only detectable if you know which judge produced which number.

MCP pinning: the prompt change that arrives from someone else’s repository

An MCP server you consume ships tool names, schemas and descriptions. Those descriptions land in your context window on every turn, so an upstream reword is a prompt edit performed by a third party — and an added tool is a capability you did not review. tool-description poisoning is the malicious version; the accidental version is more common and equally capable of moving your tool-selection rates.

The gate check is two comparisons: the server version is pinned and on your allowlist, and the hash of every tool description matches the last approved value. Fail closed. Also run it on a schedule, not just on push, because upstream releases on its own clock and your pipeline may not fire for weeks.

Egress and sandbox checks must run against the deployed config, not the repository

This is the difference between validating your intent and validating your system. A gate that reads egress-allowlist.yaml from the repo proves that the file says the right thing. It does not prove the allowlist is attached to the identity the agent actually runs as, that the console change someone made last Tuesday did not add a wildcard, or that the sandbox still refuses to write outside its workspace.

So make these checks active: in the deployed environment, attempt a connection to a destination that is not on the list and assert it fails; attempt a write outside the workspace and assert it fails; start a session, exceed the timeout, assert it dies. Slower than reading a file, and the only version that tests reality. agentops · rollout and kill-switches makes the same argument about kill switches: a control you have not exercised this release is a belief.

Where do idempotency and retry checks fit?

In the gate, as fixtures rather than assertions about code. Replay a run that fails after the write and assert the write happens once — idempotency is the property that decides whether a retry is safe, and an agent loop retries constantly.

Two cheap fixtures cover most of it: a tool that times out on first call and succeeds on second (assert one side effect, not two), and a resumed run from a checkpoint (assert the resumed run does not re-execute completed actions). These fail loudly in a gate and silently in production, which is the usual argument for moving a check left.