What 23894 adds: objectives, risk sources, lifecycle hooks
Lesson 2 of 5 in AI Risk Management and Impact Assessment: ISO/IEC 23894 and 42005.
If 23894 keeps 31000’s skeleton, where is the AI? In three annexes that do the heavy lifting — and in AI-specific guidance threaded through every process step.
Annex A: AI-related objectives. Since risk is the effect of uncertainty on objectives, 23894 first names the objectives AI puts in play: accountability, AI expertise, availability and quality of training and test data, environmental impact, fairness, maintainability, privacy, reliability, resilience, robustness, safety, security, and transparency and explainability. An organisation selects and prioritises from this list in its own context — a medical-device firm weighs safety and reliability differently from an ad-tech firm — and those chosen objectives become the yardstick every risk is measured against. You met a close sibling of this list in 42001 Annex C; the two standards deliberately rhyme.
Annex B: AI-specific risk sources. A risk source is anything with intrinsic potential to give rise to risk. Generic ERM knows fire, fraud, and FX exposure; Annex B catalogues what is distinctively dangerous about AI. Learn these seven cold — incident post-mortems almost always trace back to one or more of them.
Complexity of the environment
AI systems deployed into open-ended environments — roads, markets, public discourse — face inputs no test set anticipated. Microsoft’s Tay chatbot (2016) was engineered for playful conversation and met coordinated adversarial users instead; it was producing racist output within sixteen hours. The environment, not the code, was the risk source.
Lack of transparency and explainability
When decision logic cannot be inspected or explained, errors hide, bias evades detection, and accountability breaks. The Dutch toeslagenaffaire ran for years partly because neither affected families nor caseworkers could see why the system flagged them as fraud risks — opacity converted a model defect into a national scandal that brought down a government in 2021.
Level of automation
The more decisions flow through without human checkpoints, the faster errors propagate and the fewer chances anyone has to catch them. Australia’s Robodebt automated debt-raising at a scale and speed no manual review could match — nearly half a million unlawful debts. Automation level is a chosen risk source: the same model behind a human recommender and behind an auto-decider carries different risk.
Machine-learning-specific sources
The family of failure modes unique to learned systems: data quality problems (unrepresentative, mislabelled, poisoned training data), drift (the world moves; the model does not), and adversarial inputs (crafted examples that flip outputs). The Epic sepsis model — externally validated in 2021 at far lower accuracy than advertised, missing most sepsis cases — is a data-and-drift story wearing a healthcare costume.
System hardware issues
Models run on physical substrates: sensor degradation, compute faults, numeric precision differences between training and deployment hardware. Autonomous-vehicle perception failures often begin here — a camera blinded by low sun is a hardware risk source feeding an ML pipeline.
System life cycle issues
Risks born from process: inadequate specification, missing verification and validation, deployment without rollback plans, retirement without data disposal. Knight Capital lost $440 million in 45 minutes in 2012 from a botched deployment — no AI required; now imagine the same discipline gap wrapped around a learning system.
Technology readiness
Deploying capabilities beyond their maturity: research-grade techniques pressed into production-grade duties. Early facial-recognition deployments in policing — wrongful arrests of Robert Williams and Porcha Woodruff in Detroit — put immature technology into one of the highest-stakes decision contexts that exists.
Annex C: mapping risk management to the AI lifecycle. The third annex answers when: it relates risk-management activities to the AI system lifecycle stages you learned from ISO/IEC 22989 — inception, design and development, verification and validation, deployment, operation and monitoring, re-evaluation, retirement. The insight it encodes: risk sources have home stages. Data-quality risk is born in design and development; drift lives in operation; automation-level risk crystallises at deployment when you decide what the system may do without a human. Assess once at launch and you miss both the risks born earlier (already baked in) and later (not yet visible).
| Risk source | Inception | Design & dev | V&V | Deployment | Operation | Retirement |
|---|---|---|---|---|---|---|
Environment complexity | ● | ● | ●● | ●● | ●● | — |
Opacity / explainability | ● | ●● | ●● | ● | ● | — |
Level of automation | ●● | ● | ● | ●● | ●● | — |
ML sources: data quality | ● | ●● | ●● | — | ● | ● |
ML sources: drift | — | — | — | ● | ●● | — |
ML sources: adversarial | — | ● | ●● | ● | ●● | — |
Hardware issues | — | ● | ● | ●● | ●● | — |
Life cycle issues | ●● | ●● | ●● | ●● | ● | ●● |
Technology readiness | ●● | ●● | ● | ●● | — | — |
Interactive sorting exercise: Incident vignettes from the field. Which Annex B risk source is the primary culprit? Drag each vignette to its source.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.