The failure files I: COMPAS, Amazon, and the toeslagenaffaire
Lesson 3 of 5 in Legal Intersections, Liability, and the Failure Files.
You met these stories in foundations as cautionary tales. Here they get the treatment an expert owes them: a governance post-mortem. For each case, the same four questions — what happened, what was the root cause, which control from this domain would have caught it, and what did the failure cost. The point is not moral instruction. It is that every one of these disasters was catchable with machinery you now know how to build.
COMPAS — what happened
Northpointe’s COMPAS tool scored criminal defendants’ reoffending risk across US courtrooms. In 2016, ProPublica analyzed ~7,000 Broward County cases: among defendants who did not reoffend, Black defendants were nearly twice as likely to have been labeled high-risk as white defendants. Northpointe’s rebuttal: the scores were calibrated — a given score meant the same reoffending probability regardless of race. Both sides were arithmetically right. As the impossibility results you studied in testing and evaluation prove, calibration and equal error rates cannot coexist when base rates differ. Meanwhile Loomis v. Wisconsin (2016) upheld COMPAS-informed sentencing but required warnings — and the proprietary algorithm stayed secret from the very defendants it scored.
Root cause
Not a bug — an unexamined value choice. Northpointe optimized for calibration without anyone deciding, on the record, that calibration was the right fairness definition for a punitive context where false positives cost individual defendants their liberty. Vendor opacity then made the choice uninspectable, and courts adopted a tool validated on general populations without validation for their own deployment context.
The control that would have caught it
Documented, justified fairness-metric selection at design time — the punitive-vs-assistive analysis from the testing module, which asks who bears the cost of each error type — plus deployment-context validation and contractually secured audit rights so courts could examine what they were relying on. Every one of those is a standard artifact in the program you have built across this domain.
Consequence
COMPAS became the permanent citation for algorithmic injustice, the worked example in every fairness impossibility proof, and the reason "which fairness metric, and who chose it?" is now a due-diligence question. The tool survived; trust in unexamined risk scoring did not.
Amazon — what happened
From 2014 Amazon built an experimental résumé-screening engine trained on ten years of its own hiring decisions. Because those years skewed heavily male in technical roles, the model learned that maleness predicted success: it downgraded résumés containing the word "women’s" ("women’s chess club captain") and penalized graduates of two women’s colleges. Engineers patched the obvious terms — and could not guarantee the model had not found subtler proxies. Amazon scrapped the tool by 2018. It was never used for live hiring decisions.
Root cause
Historical labels as ground truth. Supervised learning treats past outcomes as the definition of correct; feed it a decade of biased decisions and it distills the bias into a scoring function, then launders it through mathematical neutrality. The proxy problem compounds it: remove the explicit signal and the model routes around the removal through correlated features.
The control that would have caught it
Pre-deployment disparate-impact testing — score a held-out applicant set, compare selection rates by gender, apply the four-fifths screen — would have flagged the skew in an afternoon. Upstream of that, a data-representativeness review of the training labels would have predicted it before a model existed. Amazon deserves the credit it rarely gets: internal testing caught it before deployment — which is the control working, years too late and after real money, but working. Most organizations running the same experiment would have shipped it.
Consequence
The canonical teaching case for label bias — cited in the NYC LL144 debates, the EEOC’s AI guidance, and virtually every AI hiring-law preamble since. Amazon’s reputational cost bought the industry its clearest before-the-fact warning, and vendors selling hiring AI without disparate-impact evidence now sell against it.
Toeslagenaffaire — what happened
The Dutch tax authority’s childcare-benefits fraud system risk-scored families using features including dual nationality, and treated flags as fraud findings. Tens of thousands of families — disproportionately with immigration backgrounds — were forced to repay tens of thousands of euros, driven into bankruptcy and divorce; over a thousand children were placed in care. Frontline staff who questioned the flags were overruled; affected families could not learn why they were flagged or effectively appeal for years. The Dutch DPA fined the tax administration; a parliamentary inquiry titled its report "Unprecedented Injustice"; and in January 2021 the entire Dutch government resigned. Its cousin, the SyRI welfare-fraud system, had already been struck down by a Dutch court in 2020 for violating Article 8 ECHR.
Root cause
A stacked failure — which is what makes it the richest case in the file: a discriminatory feature (nationality) no data-governance gate ever challenged; automation treated as ground truth by an institution structurally deaf to its own frontline warnings; no functioning contest channel for the accused; and no monitoring that aggregated the mounting wreckage into a signal anyone with power had to act on. Remove any one layer and the disaster shrinks; remove the contestability failure and it likely ends years earlier.
The control that would have caught it
Pick a layer, pick a control: a fundamental rights impact assessment (nationality as a fraud feature does not survive one — this case is why FRIA exists in the AI Act); data-feature governance blocking protected attributes and their proxies; meaningful human review with authority to overturn — the competence-authority-time-information test; and above all contestability with teeth, because appeal channels are also your incident-detection sensor. Thousands of unresolvable appeals were the alarm, ringing for years, connected to nothing.
Consequence
A constitutional crisis: government resignation, DPA fines, billions in compensation still being paid out, and direct influence on the EU AI Act — the deployer FRIA duty (Art 27) and the Annex III treatment of public-benefits systems trace visibly to this scandal. It is the strongest known answer to "isn’t AI governance overhead?"
Tool: Fairness Lab — Replay the COMPAS dispute yourself: drag decision thresholds for two groups with different base rates and watch calibration and error-rate parity refuse to coexist.
Key terms: calibration, base rate, label bias, proxy variable, fundamental rights impact assessment, contestability
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.