The failure files II: Clearview, Air Canada, and NEDA’s Tessa

Clearview — what happened

Clearview AI scraped billions of face images from social media and the open web — no notice, no consent, no lawful basis — and sold face-matching search to police and, for a period, private buyers. The reckoning arrived on two continents: GDPR fines in the tens of millions from France, Italy, Greece, and the UK (with the Netherlands adding a fine and Clearview simply not paying several of them, having no EU establishment); and in the US, an ACLU suit under Illinois’s BIPA — the biometric statute with a private right of action — producing a 2022 settlement that banned sales of the faceprint database to most private entities nationwide.

Root cause

A business model built on the assumption that publicly accessible means freely usable. It does not: GDPR applies to processing biometric data of EU residents wherever the scraper sits, and BIPA conditions collection on written consent. Data provenance was not a compliance gap in the product — it was the product.

The control that would have caught it

Lawful-basis analysis before collection — the first question of the data-governance module — and for any buyer, the training-data provenance question from your vendor due-diligence set. Clearview’s customers bought exposure along with the search box: procurement teams that asked "where did the faces come from, under what right?" did not become defendants or headlines. One question, asked at intake.

Consequence

Cross-continental enforcement (with the enforcement-gap lesson that fines without establishment are hard to collect), a structural US settlement, and biometric scraping cemented as a red-line category — visible in the EU AI Act’s prohibition on untargeted facial-image scraping to build recognition databases. One company’s data practice wrote a prohibited-practices clause.

Air Canada — what happened

A customer, booking travel after his grandmother’s death, asked Air Canada’s website chatbot about bereavement fares. The bot invented a policy: book now, apply for the discount within 90 days. The real policy required applying before travel. When the airline refused the refund, he sued in small claims — and Air Canada argued, remarkably, that the chatbot was "a separate legal entity responsible for its own actions". In Moffatt v. Air Canada (2024), the tribunal called the argument what it was, found negligent misrepresentation, and held the airline liable for what its own website told a customer.

Root cause

An ungrounded language model answering policy questions with no retrieval from the actual policy corpus, no output verification, and no monitoring of what it told customers — deployed at exactly the task (authoritative policy statements) where fluent fabrication is most costly. The legal theory was old; the deployment pattern was new.

The control that would have caught it

The hallucination control ladder, applied honestly: a customer-facing policy bot sits at the rung demanding RAG grounding on verified policy documents with citations, output checks, and sampled conversation monitoring. Any one of the three catches this. A pre-deployment red-team asking the bot about ten common fare policies would have surfaced the fabrication risk in an hour.

Consequence

A tiny judgment (~CA$812) and an enormous precedent: your chatbot’s words are your words, and disclaimers do not outrun specific misstatements a reasonable customer relied on. Every genAI deployment review that now asks "what happens when it misstates our policy?" is quoting this case. The "separate legal entity" defense entered the canon of things one does not say to a tribunal.

NEDA / Tessa — what happened

In 2023 the US National Eating Disorders Association disbanded its human helpline — days after its staff unionized — and moved users toward Tessa, a wellness chatbot. Tessa had been built as a scripted, rule-based program with clinical input. But its vendor, Cass, had added generative AI capabilities without NEDA’s knowledge or approval — and when users with eating disorders asked for help, Tessa produced weight-loss advice: calorie deficits, weekly weigh-ins, body measurements — near-textbook triggers for the population it served. An activist and a clinician published the transcripts; NEDA took Tessa down within days.

Root cause

A silent vendor model change that transformed a validated, bounded system into an unvalidated generative one — with no notification, no re-validation, and no output monitoring on a maximally vulnerable population. Compounded by an organizational decision to remove the human alternative before the automated one had earned trust.

The control that would have caught it

Three clauses and a monitor, all from the third-party module: a model-change notification clause with re-validation rights (the change never reaches production unannounced), output monitoring tuned to the deployment’s known red lines (any diet-advice output should have paged a human immediately), and population-sensitive pre-deployment testing — adversarial prompts a clinician would write in ten minutes. This is the cleanest known demonstration that vendor governance is safety-critical, not procedural.

Consequence

Direct potential harm to vulnerable users, national coverage, congressional letters — and the case now cited in every discussion of genAI in health-adjacent deployments. It also sharpened a labor-and-automation question the AI governance field can no longer dodge: the helpline’s humans were not a cost center to be automated away; they were the safety layer.

The wider casebook: Robodebt (Australia, 2016–2023)

Automated welfare-debt recovery by income averaging — a calculation method that was simply unlawful — raised hundreds of thousands of false debts before courts and a Royal Commission (2023) condemned the scheme; a class settlement approached A$1.8 billion. Missing control: legal-basis verification before automation — nobody with authority confirmed the core calculation was lawful — plus escalation channels for the frontline staff who knew early and were ignored. The toeslagenaffaire’s southern-hemisphere twin.

UK Ofqual A-levels (2020)

With exams cancelled, an algorithm standardized teacher-predicted grades — downgrading ~40%, and hitting bright students from historically weaker (often state) schools hardest, because the model leaned on school-level history for individual grades. Days of protests ("f*** the algorithm") forced a full U-turn. Missing controls: individual-vs-cohort fairness analysis, stakeholder consultation, and contestability designed before launch — an appeal route existed on paper but could not absorb a nation of angry teenagers. Political legitimacy is a governance requirement, not a nice-to-have.

Epic sepsis model & the Obermeyer study

Two healthcare label failures: external validation of Epic’s widely deployed sepsis predictor (2021) found sensitivity far below vendor claims (missing ~two-thirds of cases while flooding wards with alerts); and Obermeyer et al. (2019) showed a care-management algorithm used healthcare cost as the label for healthcare need — since less money is spent on Black patients at equal illness, the proxy label systematically understated their need. Missing controls: independent local validation before clinical deployment and label-choice review — the question "is the target variable actually the thing we care about?" from the testing module.

Apple Card / Goldman Sachs (2019–2021)

Viral allegations that spouses with shared finances received wildly different credit limits triggered a New York DFS investigation. The regulator found no unlawful discrimination — but its report was scathing about something else: customer-service staff could not explain the decisions, deflecting with "it’s the algorithm". Missing control: explanation readiness — reason codes surfaced to the frontline, and staff trained to deliver them. You can pass the fairness audit and still fail the trust audit.

Mata v. Avianca and the citation fabricators (2023–)

The lawyer who filed six ChatGPT-invented precedents was sanctioned — you know the story from foundations. Its post-mortem framing: a consequential-output verification duty that no fluency can waive, now formalized in court standing orders requiring AI-use disclosure and verification certificates. The professional-responsibility version of the human gate.

NYC MyCity chatbot and Tay

NYC’s official business chatbot (2024) told employers they could take workers’ tips and fire staff for reporting harassment — government genAI confidently advising law-breaking; the city kept it live with a disclaimer, testing the Air Canada lesson in public. Microsoft’s Tay (2016) — corrupted into slurs within 16 hours of learning from Twitter — remains the founding case for deployment gating of systems that learn from adversarial publics. Missing controls, respectively: grounded answers with legal review before deployment on legal-advice territory; and an abuse-aware release plan with a kill switch that fires in minutes, not headlines.

The incident record that built the field

• 2016-05-23 — ProPublica publishes “Machine Bias” (COMPAS):

  Black defendants who didn’t reoffend were flagged high-risk at nearly twice the white rate; the vendor showed equal calibration. Both were right — the fairness impossibility theorem goes mainstream.

• 2018-02-01 — Gender Shades study:

  Buolamwini and Gebru show commercial face analysis erring up to 34% on darker-skinned women vs under 1% on lighter-skinned men — the case for disaggregated testing, made unignorable.

• 2018-03-18 — Uber ATG test vehicle kills Elaine Herzberg:

  The first pedestrian death by an autonomous vehicle — a distracted safety driver behind an automated system becomes the canonical human-oversight failure case.

• 2018-10-01 — Amazon scraps its AI recruiting tool:

  Trained on ten years of male-dominated hiring, the model learned to downgrade résumés mentioning “women’s”. Killed before deployment — the textbook label-bias parable.

• 2020-01-18 — Clearview AI exposed:

  Three billion faces scraped without consent for police search. The backlash writes itself into law: EU AI Act Art 5 bans untargeted face-scraping outright.

• 2021-01-15 — Dutch government falls over the childcare-benefits scandal:

  An algorithmic fraud system with nationality as a risk factor ruined tens of thousands of families. A cabinet resigns — the starkest proof that algorithmic harm is political.

• 2023-03-31 — Italy temporarily bans ChatGPT:

  The Garante suspends the service over GDPR concerns — first proof that existing privacy law already reaches generative AI, no new statute required.

• 2023-06-22 — Mata v. Avianca sanctions:

  A lawyer files six LLM-invented precedents; the court sanctions him. Hallucination plus unverified reliance becomes case law’s favourite cautionary tale.

• 2024-02-14 — Moffatt v. Air Canada:

  The airline argued its chatbot was “a separate legal entity responsible for its own actions.” The tribunal disagreed. Companies own their AI’s promises.