Retention in Redshift and DynamoDB

Lesson 3 of 4 in Lifecycle and Tiering.

Outside Amazon S3, "retain this for ninety days and then get rid of it" has a different mechanism in every store, and — this is the part that matters for a compliance requirement — "deleted" means something different in each of them.

Amazon Redshift has no row-level retention feature. You delete rows with SQL, and space is reclaimed by VACUUM (which automatic table optimization now runs in the background, though it can fall behind a heavy append). The practical pattern is to keep recent data in managed storage, unload older partitions to Amazon S3 and query them through Spectrum, and drop the local rows. And note the backup side: automated snapshots retain for one day by default and up to 35, while manual snapshots are kept until you delete them — so a row deleted today is still inside every snapshot taken before the deletion.

Amazon DynamoDB has time to live, and it is a feature whose honest description matters. You name an attribute and store a Unix epoch timestamp in seconds as a Number; an item whose attribute is any other type is ignored entirely, which is the commonest silent failure. AWS documents the deletion as happening "typically within a few days" of expiry, so expired items remain readable in the meantime and your application must filter them out of Query and Scan results itself. The deletes consume no write throughput, appear in DynamoDB Streams as service deletions rather than user deletes, and remove the item from its indexes — and with global tables the originating delete is free while each replicated delete consumes replicated write capacity.

The open table formats are the subtlest. A DELETE on an Iceberg table removes rows from the current snapshot; the data files that held them are still referenced by earlier snapshots, so the bytes are still on disk and still readable by time travel. They are only gone once snapshots have expired and unreferenced files have been removed — which is why on Amazon S3 Tables the snapshot-retention default of 120 hours is the floor on how quickly a delete becomes real.

Two more mechanisms belong beside retention, because they are what a stem reaches for when it says "recoverable".

Point-in-time recovery in DynamoDB is a switch, not a schedule: once on, the table can be restored to any second in the retention window — 35 days by default. That covers the failure mode scheduled backups handle badly, which is a bad deployment corrupting rows for forty minutes before anyone notices. Two properties decide answers: the restore always creates a new table, so recovery is a cutover rather than an in-place rewind; and it complements on-demand backups rather than replacing them, because long compliance retention is their job.

AWS Backup centralises backup policy across services — DynamoDB, Aurora and RDS, EBS, EFS, Amazon Redshift, S3 and more — into backup plans with schedules, lifecycle to cold storage, and cross-Region or cross-account copies. The reason it appears in a data-engineering question is governance rather than mechanism: one place to prove that a retention policy exists and was applied, plus vault lock to make the backups themselves immutable.

And the honest note on all of it: retention and erasure are different requirements. "Keep seven years" is satisfied by a cheap storage class. "Erase this person's records on request" is satisfied only by knowing every store, snapshot, backup, replica and table-format file the data reached — which is why the audit and privacy work is not administrative overhead. A deletion you cannot describe mechanically is a deletion you cannot certify.

Interactive sorting exercise: Each card is a requirement. Match it to the mechanism that actually satisfies it — the point being that “delete old data” has five different implementations with five different guarantees.

Key terms: DynamoDB TTL, Point-in-time recovery (PITR), Redshift snapshot, Snapshot (table format), AWS Backup

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.