The failure taxonomy: which control would have caught it?
Lesson 5 of 5 in Legal Intersections, Liability, and the Failure Files.
Line the cases up and the noise falls away: a dozen scandals, ten failure patterns, and every pattern maps to a control taught in this domain. This is the synthesis worth memorizing — not the case details, but the mapping. When you assess a new system, you are not predicting novel disasters; you are checking for the same ten holes.
The taxonomy: proxy discrimination (nationality, cost-as-need, women’s-college vocabulary), wrong label or objective (Obermeyer, Amazon), no pre-deployment testing (Rite Aid, Epic-in-your-hospital), vendor opacity and silent change (COMPAS, Tessa), automation bias and hollow oversight (toeslagenaffaire’s overruled caseworkers), no contestability (toeslagenaffaire, Ofqual), no monitoring (Air Canada’s unread transcripts), no legal basis (Robodebt, Clearview), misrepresentation by AI (Air Canada, MyCity), and no incident response (Tay’s sixteen hours; every appeal that vanished into the Dutch tax authority).
| Case | Primary failure | The control that catches it | Taught in |
|---|---|---|---|
COMPAS | Unexamined fairness-metric choice + vendor opacity | Documented metric selection; deployment-context validation; audit rights | Testing & evaluation; Third-party |
Amazon hiring tool | Historical label bias, proxy features | Data-representativeness review; pre-deployment disparate-impact testing | Data & documentation; Testing & evaluation |
Toeslagenaffaire | Discriminatory feature; hollow oversight; no appeal | FRIA; data-feature governance; meaningful review; contestability | Impact assessments; Monitoring & incidents |
Clearview AI | No lawful basis for collection | Lawful-basis analysis; vendor provenance due diligence | Data & documentation; Third-party |
Air Canada chatbot | Ungrounded genAI stating policy; no monitoring | RAG grounding + citations; output monitoring; pre-launch red-team | Third-party & frontier |
NEDA Tessa | Silent vendor model change on a vulnerable population | Model-change notification + re-validation; population-sensitive testing; output monitoring | Third-party & frontier |
Robodebt | Automating an unlawful calculation; ignored frontline warnings | Legal-basis verification at intake; escalation channels | Inventory & intake; Operating model |
Ofqual A-levels | Cohort features deciding individual outcomes; no absorbable appeal route | Individual-fairness analysis; stakeholder consultation; contestability at scale | Testing & evaluation; Monitoring & incidents |
Obermeyer / Optum study | Cost proxy for medical need | Label-choice review at design | Testing & evaluation; Data & documentation |
Apple Card investigation | No frontline explanation capability | Reason codes surfaced to staff; explanation readiness | Testing & evaluation |
Interactive sorting exercise: Forensics drill: drag each failure signature to the control family that would most directly have caught it.
Tool: AI Incident Tabletop — Close the loop: run a live incident — containment, comms, and regulatory clocks — and see how many taxonomy holes your response plan covers.