What Leaks in Use
Lesson 2 of 3 in Data Leakage and Memorization.
Extraction is the dramatic direction of leakage. The everyday direction runs the other way: everything you put in a prompt travels to whoever serves the model. The System prompt, the user’s message, every document your RAG pipeline stuffs into the Context window — all of it crosses your boundary in plaintext (TLS protects the wire, then terminates at the provider’s edge). From that point, what happens to your data is decided by the provider’s architecture and the terms you accepted, not by anything your code enforces.
Two questions matter, and they are different kinds of question. Can the provider see, log, retain, or train on your prompts? Mechanically yes — the serving stack processes plaintext by necessity. May they, and do they? That is policy: retention windows, training-on-inputs defaults, abuse-monitoring review, subprocessor lists. Enterprise tiers, opt-outs, and data-processing terms exist precisely to narrow the gap between the two — but they only help if someone on your team has actually read the terms for the exact service and tier you call. (What those terms must say for your jurisdiction and sector is contract and privacy law — the AI Governance Academy’s territory, not this site’s.)
Where a prompt can come to rest
- User input
A message that may already contain PII, credentials pasted by accident, or client-confidential material.
- Your app assembles the prompt
System prompt + retrieved documents + conversation history + the user’s text. Leak surface you own: application logs, traces, crash reports, and analytics that capture the assembled prompt.
- Retrieval joins the party
RAG pulls internal documents into the context — data that was access-controlled at rest is now inline text in a request. Leak surface: your vector database and its own logs, plus everything downstream.
- Request crosses your boundary
Over the public internet by default; over private connectivity (PrivateLink, Private Link, Private Service Connect) if you configured it. Network path is a mechanism you choose.
- Provider serving stack
Leak surfaces you rent: request/response logs, retention for abuse monitoring, human review queues, caches, subprocessors. Scoped by your tier’s data-processing terms — a policy surface.
- Model inference
If the provider trains on inputs (a tier-dependent policy), today’s prompt can become tomorrow’s memorization risk — lesson one, fed by lesson two.
- Response returns
The reply lands back in your logs, eval sets, fine-tuning datasets, and screenshots. Outputs derived from sensitive context are sensitive too.
- Data now lives in every hop that stored it
Deleting a conversation from the UI does not delete it from logs, caches, backups, or datasets built from them.
The uncomfortable audit finding in most LLM products is not the provider — it is the product’s own plumbing. PII in context becomes PII in logs: prompt-logging middleware, distributed traces, LLM-observability dashboards, cached responses, Golden set examples harvested from real traffic, screenshots in bug tickets. Each copy has its own retention, its own access list, and its own backup schedule. A prompt is not one artifact; it is a family of artifacts scattered across systems that were never in the privacy review.
And one line for the adjacent risk: give the model tools — browsing, email, code execution — and leakage stops being passive, because injected instructions can actively exfiltrate whatever sits in context. That agent-side kill chain is the sister AI Agent Academy’s beat; here it is enough to know the context you assemble is the blast radius.
| Consumption shape | Where prompts go | Retention & training on inputs | Network path | Your control points |
|---|---|---|---|---|
Consumer chat app | To the provider, under consumer terms — often the broadest defaults | Varies by product; opt-outs may exist and defaults change — nobody on your side reads the terms unless you make it a policy | Public internet, provider-managed everything | Almost none technically. Usage policy and training for staff — what employees paste is the control |
API / managed cloud platform | To the platform, under business terms you selected | Business tiers commonly commit to bounded retention and to not training on your inputs — verify per service and tier, in writing | Public endpoint by default; private connectivity (PrivateLink / Private Link / PSC) where documented | Tier and terms selection, private endpoints, region choice, your own logging config, contractual review |
Self-hosted weights | Nowhere — prompts stay inside your infrastructure | You set retention, and you inherit the whole obligation: your logs are now the leak surface | Your VPC, your network controls | Everything — plus the full serving, patching, and capacity burden that comes with it |
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.