Privacy, safety, and security
Lesson 4 of 6 in Why AI Needs Governance: A Taxonomy of Harms and Risks.
Three harm families share a lesson: AI does not just create new harms — it takes old ones (surveillance, product defects, fraud) and removes the friction that kept them rare.
Privacy first. Machine learning is hungry, and the cheapest food is other people’s data. Clearview AI scraped over 30 billion face images from social media and the open web, built a face-search engine, and sold it to police — no consent, no notice. Regulators in France, Italy, Greece, the UK, and Australia issued penalties and deletion orders; a 2022 Illinois BIPA settlement barred sales to most US private companies. The deeper lesson: “publicly available” is not “free to use” — a principle now written into cases, statutes, and every data-protection authority’s guidance on scraping.
Beyond collection, models create three inference-shaped privacy harms with no pre-AI equivalent: memorisation (LLMs can regurgitate training data verbatim — researchers extracted personal contact details from deployed models), attribute inference (predicting sexuality, pregnancy, health status from behaviour that never mentioned them — the mechanism behind the famous Target pregnancy-prediction story), and re-identification (“anonymous” data re-linked to people by cross-referencing patterns). Each defeats the classic consent model: you cannot consent to an inference you never imagined.
Safety is where failure meets flesh. In March 2018, an Uber ATG test vehicle in autonomous mode struck and killed Elaine Herzberg in Tempe, Arizona — the system classified her as a vehicle, then a bicycle, then “other”, never braking in time, while the safety driver watched a phone. The NTSB’s verdict spread responsibility across the whole lifecycle: an inadequate perception design, a deliberately disabled emergency-braking system, and a hollow oversight arrangement. It remains the canonical case study that “a human is in the car” is not a safety case.
Generative systems added a quieter safety channel: unfit advice at scale. In 2023 the US National Eating Disorders Association replaced its helpline staff with a chatbot, Tessa — which was found recommending calorie deficits and weigh-ins to callers with eating disorders, and was pulled within days. Lawsuits over AI companions and minors have since pushed the same question into court: what duty of care attaches to a system that talks to vulnerable people about their lives?
Security cuts both ways: attacks on AI and attacks with it.
Attacks on AI: adversarial examples
Inputs crafted to fool a model while looking normal to humans — stickers that make a stop sign read as a speed limit, patterns that defeat face recognition. They exploit the gap between how models and humans perceive, and they matter most wherever a model gates something valuable: content filters, fraud checks, identity verification.
Attacks on AI: data poisoning
Corrupting the training data so the model learns an attacker-chosen behaviour — a backdoor that activates on a trigger phrase, a classifier taught to ignore one kind of fraud. Poisoning is the security face of the data-provenance problem: if you cannot trace what went into training, you cannot rule out that someone salted it.
Attacks on AI: prompt injection and jailbreaks
Prompt injection hides instructions in content an LLM will process — a webpage, an email, a résumé — hijacking systems that act on that content. Jailbreaks talk a model past its safety training. Both are unsolved in the general case, which is a governance fact: any architecture that lets an LLM read untrusted content and take actions needs controls that assume injection will sometimes succeed.
Attacks with AI: deepfake fraud and voice cloning
In early 2024, an employee of the engineering firm Arup joined a video call with what appeared to be the CFO and colleagues — every participant a deepfake — and transferred about US$25 million to the fraudsters. Voice-cloning scams now impersonate family members and executives from seconds of sampled audio. The friction that once made impersonation fraud rare — the difficulty of faking a face and a voice — is gone.
Attacks with AI: capability uplift (cyber, bio/chem)
The frontier-model concern: does the system meaningfully help a non-expert do serious harm — write working exploit code, plan a biological or chemical attack? This dual-use question drives frontier-model evaluations at AI safety institutes, the EU’s systemic-risk obligations, and US frontier-model rules. The honest current answer is “uplift is measurable and growing”, which is why pre-release capability evaluations became standard practice.
Key terms: Clearview AI, memorisation, prompt injection, data poisoning, deepfake, dual-use
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.