Criteria, identification, and treatment

Lesson 3 of 5 in AI Risk Management and Impact Assessment: ISO/IEC 23894 and 42005.

Risk criteria are where AI risk management earns its keep — or quietly fails. A classic enterprise likelihood-consequence matrix scores consequence in money, downtime, and reputation: loss to the organisation. 23894 pushes the criteria wider, and it has to. An AI system can be a commercial triumph while harming the people it scores. If your consequence scale cannot express ‘wrongful arrest’, ‘denied benefits’, or ‘population-scale misinformation’, those harms will be systematically invisible to your risk process — not through malice, but through arithmetic.

So AI risk criteria need at least three properties. Consequence scales that include harm to individuals, groups, and society alongside organisational loss. Likelihood scales that handle patterned uncertainty — an 89%-accurate model is not ‘low likelihood of error’; it is a guaranteed 11% error rate whose distribution across subgroups matters more than its average. And explicit acceptance thresholds with named owners — someone with authority decides what residual risk the organisation carries, and signs.

Identification: four techniques that find AI risks generic brainstorms miss. 23894’s guidance and practice converge on a toolkit — use several, because each has a characteristic blind spot:

Lifecycle walkthrough

March the system through every 22989 stage — inception to retirement — asking Annex B’s question at each: which risk sources are active here? Systematic and auditable; the natural backbone of your risk register.

Blind spot: it inherits its imagination from the annexes. Novel, system-specific risks (a feedback loop unique to your marketplace) need the other techniques.

Threat modelling

Borrowed from security: enumerate adversaries, capabilities, and attack surfaces. For ML pipelines that means poisoning of training data, evasion at inference, model extraction via APIs, prompt injection for LLM systems.

Blind spot: it finds attackers, not accidents. Drift and label bias have no adversary and will not appear in a threat model.

Failure mode analysis

FMEA-style: for each component (data ingestion, features, model, thresholds, human handoff), ask how it fails, how likely, how severe, and whether anyone would notice. The ‘would anyone notice?’ column is gold for AI — silent failure is the field’s signature.

Blind spot: component focus can miss emergent, whole-system behaviour — the model is fine, the workflow around it is the hazard.

Red teaming

Give skilled people licence to make the system misbehave: jailbreaking LLMs, probing bias with synthetic personas, stress-testing edge cases. The only technique that reliably surfaces surprises — which is why frontier-model rules (and the EU AI Act’s systemic-risk regime for GPAI) mandate forms of it.

Blind spot: coverage is only as broad as the red team’s creativity and time budget. A clean red-team report proves the presence of effort, not the absence of risk.

Treatment: four verbs, and a document. Once evaluation says ‘not acceptable’, ISO 31000 offers a small permanent vocabulary. Avoid — do not do the thing: decline the use case, kill the feature. Real organisations do choose it: Zillow shut down its entire iBuying business in 2021 after its price-forecasting models lost hundreds of millions in a shifting market — avoidance, chosen late. Modify — change likelihood or consequence with controls: retraining, guardrails, human oversight checkpoints, input filtering, confidence thresholds that route low-certainty cases to people. Share — allocate risk to parties better placed to carry it: supplier warranties, insurance, contractual duty splits (A.10.2 again). Retain — accept the risk, knowingly, with a record.

Whatever you choose, the arithmetic never reaches zero. What remains after treatment is residual risk, and the standard’s demand is not that it vanish but that it be estimated, documented, and formally accepted by someone with authority. An unwritten residual risk is a retained risk nobody agreed to retain — the purest form of governance debt.

Choose the treatment: a biased screening model, three ways out

Interactive decision tree — outcomes:

  • Avoidance — the underused option

    Legitimate and sometimes right: decline the use case, keep human screening, revisit when data and methods mature. Avoidance is what Zillow eventually chose for iBuying and what several police forces chose for live facial recognition. The mark of a real risk process is that ‘no’ is a reachable verdict.

  • Silent retention — the classic failure

    Treatment reduced the risk; it did not erase it. Deploying without estimating and formally accepting the residual gap means the organisation retained a discrimination risk without deciding to — indefensible in front of an auditor, a regulator, or a court.

  • Sharing misunderstood

    Risk sharing moves money, not accountability. The indemnity may cushion legal costs, but applicants are still harmed, regulators still attribute the discrimination to you, and your fairness objective is still missed. No contract converts an ethical and legal risk into a purely financial one.

  • Textbook treatment

    Modify to shrink the risk, share the financial layer, retain the documented residual with named ownership and monitoring conditions. Four verbs combined deliberately — this composite is what mature AI risk treatment actually looks like.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.