Training-Data Extraction

Lesson 1 of 3 in Data Leakage and Memorization.

An Large language model (LLM) is trained to predict the next Token across its corpus. Mostly that pressure produces generalization — the model learns patterns, not pages. But the same objective rewards storing some sequences outright: if a string appears often enough, or is distinctive enough, the cheapest way to predict its continuation is to have effectively memorized it. Memorization is not a bug bolted onto training; it is a side effect of the objective itself.

For years this was a theoretical worry. Carlini et al. (2020) made it concrete: using only query access to GPT-2 — the same interface any user has — they recovered hundreds of verbatim sequences from its training data, including names, phone numbers, email addresses, IRC conversations, and code. No weights, no gradients, no insider access. The recipe: prompt the model many ways, sample a large pool of outputs, then filter for candidates the model treats as suspiciously familiar. That filtering step is a form of Membership inference: deciding, from the model’s behavior alone, whether a piece of text was in its training set. The same idea powers Contamination checks in evaluation — one mechanism, two uses.

This is the risk the OWASP list files under LLM02, Sensitive Information Disclosure: what went into training can come back out, and Training data extraction is the attack class that makes it happen on purpose.

A training-data extraction attack (schematic, after Carlini et al. 2020)

  1. Query access to a deployed model

    The published attack was black-box: ordinary prompts in, sampled text out. Nothing more is required.

  2. Generate a large pool of samples

    Prompt the model many ways — including with snippets likely to precede interesting content — and sample many continuations.

  3. Rank samples by memorization signals

    Sequences the model assigns unusually high likelihood — especially relative to a reference such as a second model or a compressor — are candidates for memorized text. This is the membership-inference step.

  4. Verify candidates against source data

    The researchers confirmed hits by searching the model’s (public) training data. A real attacker verifies against the web or their own target data.

  5. Verbatim training text recovered

    Confirmed recoveries from GPT-2 included personal contact information that appeared in the training corpus.

How much a model memorizes is not fixed — it moves along three axes. Carlini et al. (2022) quantified them across the model families they studied: memorization grows with model scale (bigger models memorize more of the same corpus), with duplication (the more copies of a sequence in the training data, the more extractable it becomes), and with prompt context (give the model a longer verbatim prefix of the target, and it completes the rest more readily). Each relationship was measured as consistent and predictable in their setting — which turns “could our data be extracted?” from speculation into something you can estimate from how your data appears in a corpus.

Read those axes as a defender and the priorities write themselves. Scale you don’t control — models keep getting bigger. Prompt context is attacker-controlled. The axis a training pipeline controls is duplication — and that is exactly where the best-evidenced mitigation lives.

Membership inference deserves a moment on its own, because it is the quieter half of the threat. Extraction asks the model to produce training data; membership inference only asks whether a specific candidate was in the training set — typically by checking whether the model treats it as more familiar (higher likelihood, lower loss) than it treats comparable unseen text. That weaker question can still be a breach in itself: confirming that a person’s records were in a clinical model’s training data discloses something about that person, even if not one word is extracted. The same signal is what auditors use constructively, to test models for PII exposure and benchmarks for contamination before anyone hostile does.

Memorization versus generalization — why the rare and the repeated leak first

Training pushes a model toward whatever most cheaply reduces loss across the whole corpus. For common patterns — grammar, idiom, code syntax — the cheap solution is a general rule, because one rule pays rent on millions of examples. For a string that recurs verbatim many times — a license header, a boilerplate disclaimer, a widely mirrored article — the cheap solution flips: storing the sequence outright reduces loss on every copy at once. Duplication effectively multiplies the gradient signal for rote storage, which is why the duplication axis in Carlini et al. (2022) is so steep, and why deduplication attacks the problem at its root.

The dangerous edge case is the distinctive one-off. Carlini et al. (2020) focused on what they called eidetic memorization — text recoverable even though it appeared in very few training documents. A unique string (one person’s contact block, a private key, an unusual sentence) faces no competing continuations at training time: if the model learns it at all, it learns it exactly. Whether a given one-off gets memorized is not something current theory predicts reliably — the honest claims are directional. Rare-but-distinctive and heavily duplicated strings sit at the top of the risk list; well-mixed, paraphrased, unremarkable text sits at the bottom. If your threat model includes secrets in training data, assume some fraction is recoverable and measure, rather than assuming the model “only learns patterns.”

Key terms: Training data extraction, Memorization, Membership inference, Deduplication, Personally identifiable information (PII)

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.