When to call the governance function

Trigger 1 · The agent’s actions affect a person’s access to something consequential

Employment and pay, credit and insurance, housing, education, benefits, healthcare, policing, immigration — and the softer equivalents: which customers get a refund, whose account is suspended, who is offered which price. If an action changes what a specific human gets or is allowed to do, you are in the zone regulators care about most, whether or not the humans are your own employees.

Trigger 2 · You are inside a regulated sector

Financial services, healthcare and medical devices, insurance, pharmaceuticals, aviation, energy, legal services, children’s services, public administration. These sectors have their own supervisory regimes that apply independently of any AI-specific instrument — model risk management, clinical safety, record-keeping, professional duties. "The AI rules do not cover us yet" is not an answer when the sector rules already do.

Trigger 3 · Biometric or special-category personal data — including inferred

Faces, voices, fingerprints, gait; health, ethnicity, religion, political opinion, union membership, sexual orientation. The trap specific to agents is inference: a model that reads free-text support messages and tags "likely pregnant" or "probably in financial distress" has created special-category data your system architecture never planned to hold. If a field could be inferred, treat it as processed.

Trigger 4 · Personal data crosses a purpose, boundary, or jurisdiction

Data collected for support now feeding a sales agent. A retrieval corpus that pulls in HR documents. A new region served by the same deployment. A model, vector store, or tool provider in another country. Traces of EU customer conversations landing in a US observability tenant. Agents are unusually good at quietly widening data flows, because widening the corpus is often just a config change.

Trigger 5 · Irreversible external effect without an effective human gate

Money moved, a contract or legal commitment made, a message published or sent outside your boundary, a safety-relevant setting changed. The word doing the work is effective: a gate whose reviewer lacks the context, the time, or the standing to refuse is not oversight, and the blast radius is whatever the tool can reach.

Trigger 6 · The use resembles a category that is off-limits or tightly restricted

Scoring individuals across contexts, inferring emotions in workplace or education settings, deliberately covert influence on behaviour, indiscriminate biometric identification. These categories recur in every discussion of prohibited or heavily restricted use.

Do not self-assess this one, in either direction. Resemblance is a stop-and-ask signal, not a verdict — and the difference between an allowed and a forbidden variant of the same feature is often a legal distinction an engineer has no way to spot.

Trigger 7 · The agent presents itself to people, or produces content that will pass as human-made

Public-facing conversational agents, outbound messages, generated media. Transparency-style duties — telling people they are interacting with a machine, marking synthetic content — are among the most widely adopted expectations across jurisdictions, and they are cheap to implement at design time and awkward to bolt on after your brand voice ships.

Trigger 8 · A capability change to an agent that was already reviewed

A new tool. A new data source in the corpus. An autonomy tier raised from act with approval to act and report. A new market. A new MCP server added to the toolset. This is the trigger engineers miss most often, because it feels like ordinary iteration — and it is exactly the moment the reviewed system stops being the running system. Your version tuple already detects it; wire the tuple change into the review, not just the eval gate.

Do you need governance before you ship this?

Interactive decision tree — outcomes:

• Proceed

  No trigger fired and you are inside a reviewed envelope. Ship it — and keep emitting the evidence from lesson 2, because the envelope is only as current as your last change. Re-enter this tree the moment a tool, data class, autonomy tier, or market changes, which your version tuple already tells you. Log the fact that you ran this assessment and what it concluded; a two-line note now is the record that saves an argument later.

• Proceed with governance review

  Keep building, but open the conversation now rather than at launch — and bring the system card, not the question. What makes this a short meeting: the capability summary, the tool inventory with permission scopes, the autonomy tier per action type, the data classes in and out, the oversight design with its measured review metrics, the stop mechanism and its tested time-to-stop, the retention and access story for traces, and a named owner.

  Ask for two things in return: which obligations they believe attach, and which design choices would change the answer. That second question is what turns review into engineering input instead of a verdict.

• Stop and escalate

  Do not ship, and do not resolve this inside the engineering team. Two situations land here. Either the use resembles a category that may be off-limits — where a wrong self-assessment is unrecoverable and the correct move is to ask before building further — or the agent produces consequential effects on people with no human able to refuse in time, which is the oversight expectation that appears in nearly every framework.

  The second case usually has an engineering answer: drop the affected action to act with approval, or narrow scope to the population where autonomy is defensible, and re-enter with evidence. That is a design change made in a week, not a launch reversed in public.