Privacy, residency, and deletion

Lesson 4 of 4 in Audit Logs and Privacy.

Privacy becomes a pipeline requirement the moment someone asks a question a policy document cannot answer: where is this data, how long will you keep it, and can you remove one person from it? Each of those is an engineering property, and a platform that was accreted rather than designed usually fails the third.

Residency. Customer data in a Region stays in that Region unless you move it — and a data pipeline has many ways to move it: a cross-Region replication rule, a snapshot copy, a global table, a cross-Region datashare, a Firehose destination in another Region, a disaster-recovery job nobody reviewed. Intention is not a control, so residency is implemented preventively: a service control policy on aws:RequestedRegion, Control Tower or declarative Region-deny controls, and Config rules reporting resources outside the allowed set. Two caveats become questions. Global services have a home Region — IAM and Organizations data lives in us-east-1, and some services store configuration metadata centrally even where the data plane is Regional, so "no metadata may leave the EU" is a stricter claim than "no data may leave the EU" and is not always satisfiable. And residency is not sovereignty, which adds requirements about who may operate and access the infrastructure.

Retention is two-sided and most platforms implement one side. Keeping data long enough is remembered. Not keeping it longer than necessary shows up in a privacy audit, in a discovery request that sweeps ten years of records nobody needed, and in the bill. Enforce it automatically: S3 Lifecycle expiration including noncurrent versions and expired delete markers, CloudWatch Logs retention per group, Kinesis and MSK retention periods, AWS Backup plan lifecycles, and a table-format expiry job for warehouse data. And set it with the obligation in mind — an obligation of seven years means a design target of seven years and a margin, not "indefinitely, to be safe".

Deletion requests are where the platform's shape is tested, because deleting a row from the warehouse is the easy part. The question is where else the value went: the raw landing zone, the curated partition, a materialised view, a Redshift copy, an OpenSearch index, a BI extract, the application and CloudTrail logs that quoted the identifier, the token vault, and every backup and snapshot taken since.

Four design decisions make this tractable rather than heroic, and the first is the one with the most leverage:

1. Tokenise at ingest. If the identifier was replaced by a token at the door, deleting the vault entry severs the link everywhere at once — in every copy, derivative, index and backup, without finding them. This is the single highest-return privacy decision available to a data engineer.
2. Use a table format with row-level deletes. A Hive-style table over Parquet has no notion of deleting one row: removing a record means rewriting the file it lives in, which for a day-partitioned dataset can mean rewriting gigabytes to erase one person. Apache Iceberg supports DELETE FROM and UPDATE, tracking the change in metadata and delete files so only the affected data files are touched — issued from Athena, Amazon EMR, AWS Glue or Amazon Redshift. Pair it with periodic compaction, since accumulated delete files slow reads.
3. Record lineage well enough to enumerate derivatives. "Which tables were built from this one" is unanswerable on most platforms, and it is the question a deletion request forces.
4. Keep a retention schedule, so the set of places to search is bounded rather than growing.

And the tension to state honestly, because an exam question can turn on it: a record under Object Lock in compliance mode cannot be deleted before its retain-until date by anyone, including the account root user. A legal-hold obligation and an erasure obligation therefore have to be reconciled in the retention design — which data is subject to which regime, held in which bucket — rather than at request time, when there is no mechanism left to reconcile them with.

Governance is the layer above all of this, and the exam treats it as recognition: which service holds which piece.

Classification turns security from a per-dataset negotiation into a lookup. A workable scheme has few levels, an unambiguous assignment rule, and a named control set per level — who may grant access, whether a customer managed key is required, whether cross-Region copying is allowed, what retention applies, whether masking is mandatory outside a named role. The label has somewhere to live and something to enforce it: tags on buckets and tables, LF-Tags in the catalog driving Lake Formation grants, DataZone metadata forms, and Config rules or SCPs keyed on the tag value. Macie is how you check the label is true — a bucket tagged internal that automated discovery scores highly for personal data is a classification failure, and that comparison is the most useful report a governance programme produces.

Amazon DataZone is the human layer where Lake Formation is the enforcement layer. Domains organise resources by business area; projects are the working unit whose members share assets. A producer brings assets into a project inventory, curates them with business names, descriptions, business glossary terms and metadata forms, and then publishes them — until published, an asset is discoverable only inside its own project. Consumers search the catalogue and subscribe, raising an approval request to the owner. The part that makes it more than documentation: for AWS Glue tables managed by Lake Formation and for Amazon Redshift tables and views, an approved subscription is fulfilled automatically — DataZone makes the underlying grant. For every other asset type it emits an event instead, which you wire to your own fulfilment. Two operational notes: only the latest version of an inventory asset can be published, and an updated asset must be published again, which is the usual cause of a catalogue drifting out of date.

The owner and the steward are what make any of it resolvable. Every mechanism here assumes someone can answer "may this team have this data"; without a named owner per dataset the answer defaults to whoever is least likely to say no, and subscription queues stall. On AWS the role is expressed concretely rather than on an org chart — project ownership in DataZone, a data lake administrator or grantor in Lake Formation, an owner tag on the bucket and table, the approver on the subscription. The point is not the title but that the access decision has an address.

A last note for completeness: AWS Audit Manager collects evidence continuously against a framework and produces assessment reports, which is the mechanism behind "we can demonstrate this control was operating for the whole period" rather than "it is configured correctly today". And AWS Config with conformance packs is what reports the drift in between.

Interactive sorting exercise: Sort each control by the privacy duty it actually discharges. Several are good practice that does nothing for the duty people assume — putting them in the right bucket is the whole exercise.

Key terms: Data residency, Deletion request (right to erasure), Retention schedule, Row-level delete (open table formats), Amazon DataZone, DataZone subscription, Data classification, Data steward (and data owner)

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.