Synthetic data, poisoned data, and data you must forget

Lesson 3 of 5 in Data Governance and Documentation Artifacts for AI.

Three frontier problems complete the data-governance picture, and each one breaks an intuition inherited from ordinary records management.

Synthetic data is a governance tool with its own failure modes. Generating artificial records that mimic real distributions can dodge privacy constraints (train on synthetic patients, not real ones), balance rare classes (synthesise the fraud cases you lack), and enable sharing where real data cannot move. But three risks demand documentation discipline. Privacy leakage: a generator overfitted to its source can reproduce near-copies of real individuals — synthetic is a spectrum, not a switch, and privacy claims need testing (membership-inference resistance), not assertion. Fidelity drift: synthetic data inherits the biases of its source and adds artefacts of the generator; models trained on it need validation against real held-out data. Recursive degradation: when model outputs become the next model’s training data, quality and diversity decay across generations — model collapse, demonstrated by Shumailov et al. in Nature (2024). The tails of the distribution vanish first: rare cases, minority language, unusual patterns — precisely the cases fairness depends on.

Model collapse: the recursive-training feedback loop

  1. Generation 0: trained on human data

    Full distribution: common cases, rare cases, minority dialects, edge patterns. This is the last generation with a clean provenance story.

  2. Outputs flood the web

    Model outputs are published at scale — articles, images, code — largely unlabeled as synthetic. The public corpus is now a mixture.

  3. Next model scrapes the mixture

    Without provenance metadata distinguishing human from synthetic content, the training pipeline cannot filter what it ingests.

  4. Distribution narrows

    The model over-samples its predecessor’s high-probability outputs. Tails erode first: rare events, minority voices, unusual styles fade from the learned distribution.

  5. Generation N: collapse

    Outputs converge toward bland, repetitive, homogenised content — Shumailov et al. (Nature, 2024). The defence is provenance at internet scale: labeling synthetic content (C2PA, watermarking, AI Act Art 50(2) marking) and curating verified-human corpora.

Poisoning makes the training set an attack surface. Data poisoning plants crafted examples so the trained model misbehaves on the attacker’s trigger — and research has shown that poisoning a tiny fraction of a web-scale corpus can implant backdoors. Treat datasets like software dependencies, because that is what they are: verify checksums and signatures on third-party datasets, pin versions, record the source per shard, and prefer curated snapshots to live scrapes. The supply-chain thinking your security team applies to packages applies verbatim to training data — MITRE ATLAS catalogues these techniques alongside conventional ML attacks, and the testing module covers the attack taxonomy in depth.

And then the hardest question: deleting data a model has already learned. GDPR’s right to erasure meets an awkward fact — a trained model is not a database. Removing the record from the training store does not remove its influence from the weights, and in some cases the model can regurgitate memorised training examples verbatim. This is the machine unlearning problem. Exact unlearning means retraining from scratch without the data — often prohibitively expensive; approximate unlearning techniques are an active research field, not a dependable control. Operationally, the answer is architectural: minimise personal data entering training at all, keep lineage precise enough to know which models a person’s data touched, set retention policies for training snapshots, and understand the worst-case remedy is the FTC’s: delete the model.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.