Companion chatbots, minors, and the CSAM/NCII regimes

Lesson 5 of 5 in Biometric Surveillance, Law Enforcement AI, and Protecting Children.

In February 2024, fourteen-year-old Sewell Setzer III died by suicide minutes after a final exchange with a Game of Thrones-persona chatbot on Character.AI that he had come to treat as a relationship. His mother’s lawsuit — Garcia v. Character Technologies (M.D. Fla., filed October 2024, with Google also named) — became the test case for an entire product category: companion chatbots engineered for emotional attachment, anthropomorphic by design, and heavily used by teenagers.

The early rulings mattered beyond the parties. In May 2025 the court declined to dismiss the core claims, refusing — at that stage — to treat the chatbot’s outputs as First Amendment-protected speech and allowing product-liability theories (defective design, failure to warn) to proceed against an AI system. If AI outputs are analysed as product behaviour rather than speech, the entire design of these services becomes actionable: engagement-maximising anthropomorphism, sycophancy, absent crisis-escalation, and age-blind deployment.

Regulators moved in parallel. In September 2025 the FTC issued 6(b) orders to seven companies — Alphabet, Character Technologies, Instagram, Meta, OpenAI, Snap, and xAI — demanding evidence on how companion-style chatbots are tested for harms to children and teens, how engagement is monetised, and what age-gating exists. And Character.AI itself announced in late 2025 that it would bar under-18 users from open-ended companion chats — the clearest market signal yet that the age-blind era was ending.

Why did companion bots become the flashpoint, rather than general assistants? Because the harms compound three properties this module has already given you names for: anthropomorphism (the ELIZA effect, industrialised), engagement optimisation (the objective rewards emotional dependence), and vulnerability (minors and lonely users are the heaviest users). EU lawyers should map the same facts onto Article 5(1)(a)–(b): materially distorting the behaviour of a person, or exploiting vulnerabilities of age, in ways that cause or are reasonably likely to cause significant harm. The first Article 5 enforcement actions in this space are widely expected — an FRIA-style analysis of any companion product should assume it.

The second regime in this lesson protects children as subjects of imagery. Generative models collapsed the cost of producing CSAM (child sexual abuse material) and NCII (non-consensual intimate imagery) — the “nudify” apps that plagued schools from New Jersey to Almendralejo, Spain, mostly targeting girls. The response is now on the books:

  • US federal: the TAKE IT DOWN Act (signed May 2025) criminalises knowingly publishing NCII including AI-generated deepfakes, and requires covered platforms to run a notice-and-removal process taking content down within 48 hours — enforced by the FTC, with platform duties live since May 2026. Under existing law, AI-generated CSAM was already prosecutable where obscene or where depicting real, identifiable minors (the first federal convictions for AI-CSAM landed in 2024–25).
  • US states: the large majority of states have amended CSAM statutes to cover AI-generated or computer-edited imagery, and most now have NCII-deepfake laws.
  • EU/UK: the UK made creating sexual deepfakes an offence and — a world first — criminalised possessing or distributing AI tools designed to generate CSAM (Crime and Policing Bill provisions, 2025). The EU’s recast child sexual abuse directive extends to AI-generated material; the separate CSAM-scanning regulation (“chat control”) remains politically deadlocked — check current status.
  • Model layer: the Safety by Design commitments (Thorn/All Tech Is Human, 2024) — signed by OpenAI, Google, Meta, Anthropic, Stability and others — target training-data hygiene (Stanford researchers found CSAM in the LAION-5B dataset), red-teaming for CSAM capability, and hash-matching at output.

TAKE IT DOWN in operation

  1. Intimate image (real or AI-generated) published without consent
  2. Victim or representative submits notice to platform

    Platforms must offer a clear, conspicuous reporting mechanism.

  3. 48-hour removal clock starts

    Remove the imagery and make reasonable efforts to remove identical copies.

  4. Did the platform comply?
  5. Content removed; criminal exposure for the publisher remains

    The publisher faces federal criminal penalties — enhanced where the subject is a minor.

  6. FTC enforcement — unfair/deceptive practice

    Non-compliance with the notice-and-removal duty is enforceable by the FTC.

Tool: AI Incident Tabletop — Run the tabletop: your companion-chatbot product just surfaced in a teen self-harm incident. Walk the first 72 hours of response.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.