Why an agent engineer cannot skip governance

Lesson 1 of 3 in Where Agents Meet the Law.

This module is short on purpose, and it is the one module in this academy that ends by sending you somewhere else.

Here is the argument in one sentence: the thing that makes a system interesting to a regulator is not the model, it is the action. A chatbot that answers a question badly produces a bad answer. An agent that decides badly cancels an order, rejects an application, emails a customer, moves money, or changes a production system. Regulation has always concentrated where automated decisions land on people — and autonomous action is precisely the property you spent this whole course learning to build.

So the boundary is not "AI teams do models, legal does law." The boundary is that you own the mechanisms and the evidence, and the governance function owns the interpretation. If you build without knowing which mechanisms and which evidence they will ask for, you will be retrofitting a system that was never instrumented for it — the single most expensive way to arrive at compliance.

Currency and scope — read this before the next three paragraphs

Everything below is deliberately kept at the named-framework level: what each instrument is for and what shape of obligation it creates. You will find no article numbers, no dates of applicability, no statements about whether your system falls in a particular risk category, and no compliance checklists.

That is not modesty, it is accuracy. This is the fastest-moving legal area in technology: texts get amended, guidance and harmonised standards land after the text, enforcement practice arrives later still, and the same deployment can be classified differently in two jurisdictions. Anything specific enough to be useful is specific enough to be out of date or wrong for your context by the time you read it.

Current as of September 2026, and not legal advice. Check current official guidance, and check it with the people whose job that is.

Three instruments come up in nearly every agent conversation. Read them as three different kinds of thing — a law, a management standard, and a shared vocabulary — because engineers routinely mash them into one undifferentiated "compliance" blob and then argue past each other.

EU AI Act — a law, risk-tiered

The EU AI Act is binding law, and its central design idea is that obligations scale with risk of use, not cleverness of technology. The same model can sit in a barely-regulated product and in a heavily-regulated one; what moves it between tiers is the context it is deployed into and what its output does to people. A handful of uses are ruled out entirely, a defined category carries substantial obligations, and lighter transparency duties apply more broadly.

Two themes matter directly to your architecture. First, human oversight is treated as something that must be genuinely effective — a person able to understand, intervene in, and stop the system — which is a design requirement, not a paragraph in a policy. Second, obligations attach to roles (the party that builds and places a system on the market versus the party that deploys it), and an internal platform team plus a business unit deploying agents on it can easily hold different roles for the same agent.

What you must not do with this paragraph: conclude that your agent is or is not in a particular tier. Classification is a legal determination about a specific use in a specific market, made with counsel and current guidance — never inferred from a training module.

ISO/IEC 42001 — a management system

ISO/IEC 42001 is not a law and not a technical benchmark. It is a management-system standard for AI, the same family of idea as ISO 27001 for information security: define scope and roles, assess risks and impacts, set controls and objectives, keep documented evidence, monitor, audit internally, and improve on a cycle. Organisations can be certified against it by an accredited body, which is why it shows up in enterprise procurement questionnaires and customer security reviews long before any regulator calls.

The reason it matters to you is unglamorous and important: a management-system standard turns governance into repeatable process with retrievable artifacts. An auditor does not ask "is your agent good?" — they ask to see the risk assessment for this system, the control you said you implemented, the record showing it operated, and the review that caught the last gap. Everything in lesson 2 exists to answer questions in that shape.

Whether it is worth pursuing certification, and what the current edition requires, are decisions for your governance and security functions with the current published standard in hand.

NIST AI RMF — shared vocabulary

The NIST AI Risk Management Framework is voluntary — a US government framework with no enforcement behind it — and its practical value is that it gives cross-functional teams a common vocabulary for talking about AI risk. It is organised around a small set of functions, commonly summarised as govern, map, measure, manage: establish accountability, understand context and what could go wrong, quantify it, then act on it and repeat.

Voluntary is not the same as ignorable. Frameworks like this are how enterprise risk teams structure their questions, how vendor assessments get worded, and how internal policy gets drafted, so learning the vocabulary buys you a straighter conversation. When someone asks you to "map" your agent, they are asking for the inventory and context you would put in a system card; when they ask you to "measure", they mean your evals and your monitoring, not your vibes.

NIST also publishes profiles and companion material for specific technology classes; which of those currently applies to agentic systems is exactly the kind of thing to check at the source rather than recall.

Key terms: autonomy, human-in-the-loop, approval gate, trace, eval, blast radius

Notice what all three have in common, and what they have in common is the useful part: each one asks you to show your work. Documented purpose and context, an assessment of what could go wrong, controls that actually operate, records that survive the run, oversight that a human could really exercise, and a process for change. None of that is legal drafting. All of it is engineering you have already been taught.

Which leaves four beliefs that reliably get agent teams into trouble.

Myth: "Governance is legal’s problem — they will tell us if something is wrong"

They will tell you — after the architecture is frozen, usually during a customer security review or a launch sign-off, and the fix will be "instrument the thing you shipped without instrumentation." Governance asks for artifacts that can only be produced while the system runs: per-run records, approval history, test results tied to versions. You cannot backfill a trace for a decision that already happened. Being early is cheaper than being right late.

Myth: "We are internal-only, so none of this applies"

Internal changes the audience, not the effects. An agent that screens CVs, scores employee performance, allocates shifts, or flags expense claims is making consequential decisions about people who happen to be your colleagues — one of the situations regulators are most interested in, and one where employment law and works-council obligations may bite independently of any AI-specific instrument. "No external users" is a statement about your traffic, not about your risk.

Myth: "Our model vendor is compliant, so we inherit it"

You inherit facts about the model, not about your system. The vendor did not choose your tool permissions, your autonomy tiers, whose data flows into the context window, what the agent is allowed to send outside your boundary, or who reviews its actions. Those are yours, and they are where nearly all the risk of an agent lives. Vendor documentation is an input to your assessment, not a substitute for it.

Myth: "We have a human in the loop, so oversight is covered"

Only if the human can actually intervene. A reviewer approving 99.8% of actions in a three-second median is not oversight, they are a rubber stamp — and the fact that your diagram says act with approval makes the paperwork worse, not better, because you are now claiming a control that measurement contradicts. Effective oversight means the reviewer has context, time, authority to refuse, and a working way to stop the system. Measure your gates before you describe them.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.