Case: Seventy-Two Hours After the Chatbot Leaked
A retail bank’s GenAI assistant shows one customer another customer’s transactions, the screenshot goes viral before monitoring notices, and the next 72 hours expose every gap the launch review missed.
A composite teaching case: realistic fiction assembled from well-documented public patterns — not a real engagement.
The setup. Call the bank Harborview — a composite retail bank assembled from several well-documented public incident patterns, not a real institution. Harborview serves about two million customers and, nine months before the incident, had launched 'Compass': a GenAI assistant in the mobile app that answers questions about transactions, fees, and products. Architecture: a hyperscaler-hosted foundation model behind the bank’s LLM gateway, with a retrieval layer that pulls the customer’s recent transactions into the prompt context. The launch review had focused on the obvious fears — hallucinated financial advice, toxic output — and shipped with content filters, a denied-topics list, and a system prompt forbidding advice.
On a Saturday morning, a customer asked Compass why a direct debit had bounced. The answer opened normally, then continued: '…and regarding the payment to your landlord, Mr. ——, from account ending 4417…' — a name, a partial account number, and three transactions belonging to a different customer. She screenshotted it and posted it with the caption 'my bank’s AI just showed me someone else’s money.' By the time anyone at Harborview saw it, the post had 40,000 views.
The technical cause, established later: the retrieval layer cached customer context per worker process, and a concurrency bug could, under load, serve one session’s cached records into another session’s prompt. The bug was months old. What made it visible was a vendor model-version change two weeks earlier — the new version quoted retrieved context far more verbatim than the old one, which had paraphrased loosely enough that the occasional cross-contamination read as generic filler. Nobody had gated, tested, or even announced the model change internally. It had arrived as a silent platform update.
| Hour | Event | Decision | Lesson |
|---|---|---|---|
H0 | Customer posts the screenshot on a Saturday morning. | — | The clock that matters starts before you know it exists. |
H+4 | A junior social-media analyst flags the post to her manager. No monitoring alert has fired; no internal system knows anything is wrong. | Manager escalates to the duty security officer — via a personal text, because the incident tool has no category for "AI output". | Detection came from social listening, not the AI stack. Finding #1 was written in this hour. |
H+5 | Security duty officer triages it as "possible data exposure — unconfirmed." Attempts to reproduce fail: the cache bug is load-dependent. | Declare a security incident anyway, on the strength of the screenshot. | Declare on evidence of impact, not on reproducibility. Non-determinism is not exoneration. |
H+6 | War room convenes. The kill-switch question: digital channels owns the UI, platform owns the gateway, the vendor owns the model. The runbook says "the service owner may disable the service" — and three teams each think that means someone else. | Forty minutes of debate ends when the CTO, reached on a golf course, says "turn it off, my authority." | A kill switch without a named owner is a debate, not a control. Forty minutes at viral speed is expensive. |
H+7 | Compass taken offline; app shows a maintenance banner. | Keep the gateway logs and cache servers untouched for forensics. | Preserving state beat restoring service — barely. Someone had already proposed "reboot everything and see if it fixes it." |
H+9 | First scoping attempt. Prompt/response logging is ON for prompts but retrieved-context payloads were deliberately never logged — a data-minimisation decision from the launch DPIA. There is no direct record of whose data entered whose prompt. | Forensics pivots to reconstructing exposure from cache-server session IDs and gateway timing metadata. | The privacy-motivated logging gap becomes Finding #4 — and turns a two-day scoping job into four. |
H+20 | DPO rules the event a personal data breach: name + account fragment + transactions = confidentiality breach under GDPR Art 4(12). Debate over when Harborview became "aware" for the Art 33 72-hour clock. | DPO sets awareness at H+5 (triage), rejecting the argument for "when we confirm root cause." Notification deadline: H+77. | The 72h clock runs from reasonable awareness of a breach, not from full understanding of it. |
H+31 | Comms wants to wait for "complete facts" before saying anything; the post has hit the trade press. Someone in the war room cites the AI Act’s 15-day serious-incident clock as the deadline. | DPO corrects the record: Compass is not a high-risk AI system — the operative regime is GDPR, and the deadline is 72 hours, phased notification allowed (Art 33(4)). Holding statement goes out at H+34. | Know which regime governs before the incident. The wrong statute in the war room costs hours. |
H+52 | Notification filed with the supervisory authority: what is known, what is not yet known, and a committed date for the follow-up report. | Notify with honest gaps rather than wait for certainty. | Regulators have seen phased notifications; they have less patience for late ones. |
H+68 | Preliminary root cause lands: concurrency bug in the context cache, made visible by the ungated vendor model-version change at H−14 days. | The model change enters the incident record as a contributing cause — over the platform team’s objection that "the vendor did it." | A vendor’s change on your critical path is your change. If you didn’t gate it, you accepted it. |
Days four through six: the log gap. Scoping the breach — who else saw someone else’s data — took four days, and every one of those days traces back to the launch decision not to log retrieved-context payloads. The forensics team had to correlate cache-server session identifiers against gateway request timing to infer which sessions could have received a contaminated prompt, then intersect that with the model version’s deployment window. The final numbers: roughly 1,900 conversations in the exposure window, 214 with confirmed or probable cross-customer content, 31 where the exposed data was sensitive enough to trigger customer notification under Article 34.
The bitter irony was fully appreciated in the retro: the payload-logging decision had been made for privacy — minimise what you retain — and it was the single biggest obstacle to protecting the affected customers. The fix the team landed on later threads that needle: log pointers, not payloads. Record the document and record IDs retrieved into each prompt context, not the content itself. That gives you exact breach scoping in minutes without warehousing a second copy of everyone’s transactions in a log system.
Finding 1 — Detection: the monitoring watched the wrong things
Dashboards tracked latency, token spend, and content-filter hit rates. Nothing watched for cross-customer data in outputs, and nothing alerted on the distribution shift when the new model version started quoting context verbatim — a change plainly visible in retrospect in average response length and quotation rate. First outside notice beat first inside notice by four hours.
Finding 2 — Kill switch: authority was ambient, not assigned
The runbook’s 'service owner may disable' clause worked for single-team services and failed for a system spanning three teams and a vendor. Nobody doubted it should be turned off; everyone doubted they were the one entitled to do it without career risk. The cost was forty minutes of viral spread and one interrupted golf game.
Finding 3 — Change management: the vendor’s model version was nobody’s change
Harborview gated its own code through change advisory, canary deploys, and rollback plans. The vendor’s model version — the single most behaviour-determining component in the system — arrived unannounced, untested, and unrecorded. No evaluation suite ran against it; no one could later say what else it had changed. The bug it exposed was Harborview’s own, but the exposure was jointly authored.
Finding 4 — Logging: designed for debugging and data minimisation, not for breach scoping
Prompts and responses were logged; retrieved-context payloads were not, by DPIA design. The DPIA had asked 'what is the privacy cost of logging?' but never 'what is the privacy cost of not being able to scope a breach?' Incident-reconstruction needs were absent from the logging requirements entirely.
Finding 5 — Regime confusion under pressure
Smart people in a war room reached for the wrong law twice: once proposing the AI Act’s serious-incident clock for a system that is not high-risk, once arguing the GDPR clock ran from root-cause confirmation. Both corrections cost hours. The post-incident fix was a one-page 'which regime, which clock' sheet in the runbook — GDPR 72 hours; AI Act 15/10/2 days for their genuinely high-risk systems; sectoral reporting — approved by legal in advance.
The 72-hour fight, in detail. The sharpest conflict of the whole incident was not technical. Comms had a defensible instinct — every fact released early that later proves wrong becomes its own story — and argued for saying nothing until root cause was confirmed. The DPO had a statute: Article 33 requires notifying the supervisory authority within 72 hours of becoming aware of the breach, and 'aware' means a reasonable degree of certainty that a breach occurred, not certainty about how. The compromise that held: notify the authority at H+52 with explicit known/unknown sections and a committed follow-up date under the phased-notification allowance of Article 33(4); publish a short, factual holding statement; and notify the 31 high-risk-exposure customers individually under Article 34 once the scoping stabilised on day six — each with a named contact, not a mailbox.
Two things made the H+52 filing possible at all. First, the DPO had fixed the awareness timestamp early and in writing, which ended the recurring 'does the clock even run yet' relitigating. Second, the notification template had existed since before the incident — the team filled in fields instead of drafting prose at 2 a.m. The retro line that survived into the training deck: 'the statement we most regretted was the one we delayed, not any we published.'
The kill-switch path rebuilt after the incident
- Severity signal
Any source counts: monitoring alert, staff report, social listening, customer complaint. The intake no longer cares where the evidence comes from.
- Duty incident manager triages
A 24/7 rotation with "AI output incident" as a first-class category, and the screenshot test: would this look like a breach to a regulator?
- Customer data or safety implicated?
- Suspend at the gateway
One switch, at the LLM gateway, cutting all model traffic for the affected service. The duty incident manager holds explicit, documented authority — suspend first, convene after. Drilled quarterly.
- Preserve state
Freeze logs, cache contents, and the model-version record before anyone "fixes" anything.
- Convene war room + start regulatory clock assessment
DPO joins by default and fixes the awareness timestamp in writing within the first hour.
- Standard incident process
- Restore only via change gate
Compass returns behind the same evaluation gate that now fronts every vendor model change.
The regulator correspondence. The supervisory authority’s first response, ten days in, was a questionnaire, and its questions were a syllabus in what regulators actually probe: Was there a DPIA, and did it consider this failure mode? (Yes, and no — it considered model outputs, not context contamination.) What are the processor arrangements with the model vendor, and do they cover sub-processor and model changes? (The DPA was boilerplate; model-version changes were not contemplated anywhere.) Why did breach scoping take four days? (The honest answer — the logging design — went in verbatim, with the pointer-logging redesign attached as remediation.) What technical and organisational measures existed at the time, per Article 32?
Correspondence ran seven months: the initial notification, the day-six follow-up report, the questionnaire response, one video conference, and a final letter. The outcome — realistic for a case with prompt notification, honest gaps, and visible remediation — was a formal reprimand with corrective measures and deadlines, not a headline fine. The supervising case officer’s closing letter noted two things favourably: the notification arrived inside 72 hours despite incomplete facts, and the remediation programme addressed causes rather than symptoms. Inside the bank, the political ledger read differently: the chief digital officer’s GenAI roadmap was frozen for two quarters, and every AI initiative now routes through a review board that did not exist before the screenshot.
| What failed | Control built | Evidence it produces |
|---|---|---|
No output-side data screening | Sensitive-information filter on the output path at the gateway, masking account numbers and names that do not belong to the authenticated session’s customer. | Per-policy intervention metrics — every mask or block is a counted, timestamped event on a dashboard someone owns. |
No behavioural monitoring | Continuous evaluation on sampled production traffic: groundedness, verbatim-quotation rate, response-length distribution — with alerts on shift, not just on threshold. | A baseline per model version; a paged alert fired in the next vendor upgrade’s canary, three months later. |
Ungated vendor model changes | Vendor model versions treated as changes under change management: pinned versions where the platform allows, an evaluation suite that must pass, canary rollout, sign-off recorded. | A change record per model version with attached eval results — the artifact the regulator asked for and did not get. |
Logs could not scope the breach | Pointer logging: retrieved document and record IDs per prompt context, retained 12 months; payloads still unlogged. | Breach scoping rehearsed in a tabletop at 40 minutes, down from four days. |
Detection depended on luck | Social-listening feed wired into incident intake as a first-class signal; customer-support macros for reporting suspected AI output problems. | Time-to-detection tracked as a KPI with the four-hour incident figure as the ceiling to beat. |
Kill-switch ambiguity | Named duty-role authority to suspend at the gateway, documented in the runbook, drilled quarterly with a timed exercise. | Drill reports — the second drill cut suspend time to six minutes. |
The bill. In money: forensics and external incident support ran €150–250k; legal fees across seven months of correspondence roughly the same again; the engineering remediation programme — output filters, pointer logging, the model-change gate, monitoring rebuild — consumed about three engineer-quarters plus a permanent on-call rotation. Notification and customer-care costs for the 31 individually notified customers were small in cash and large in hours. Total direct cost landed somewhere in the €500–900k range, which is to say: less than one year of Compass’s projected cost savings, and nobody at Harborview finds that comparison comforting.
In time: Compass was offline six weeks — four for remediation, two waiting for the new review board to convene and approve relaunch. In political capital: the CDO’s two frozen quarters, one platform-team lead who left in the aftermath of the 'the vendor did it' argument, and a risk function that now holds a veto it occasionally uses. The one asset on that side of the ledger: the incident bought the governance team three years of budget arguments in a single weekend. Every control in the table above had been proposed, in some form, before launch. All of them had lost to the ship date.