Lifecycle and retention
Lesson 4 of 4 in Secrets, Masking, and Resilience.
Retention is a two-sided obligation and most teams only implement one side. Keeping data long enough is the side everyone remembers. Not keeping it longer than necessary is the side that shows up in a privacy audit, in a discovery request that sweeps up ten years of records nobody needed, and in a storage bill. Automatic lifecycle management is how both sides get enforced without anyone remembering.
S3 Lifecycle configurations are sets of rules, each scoped to the whole bucket or filtered by prefix, object tags or object size, containing two kinds of action. Transition actions move objects between storage classes as they age — the classic ladder from Standard through Standard-IA or the Glacier classes, with minimum-duration charges to respect on the colder tiers. Expiration actions delete them: current versions after a set age or on a specific date, and in a versioned bucket, noncurrent version expiration after a number of days (optionally keeping the most recent few) plus the cleanup of expired object delete markers. One more rule belongs in every bucket you own: abort incomplete multipart uploads after a few days. Failed uploads leave parts that are billed as storage and are invisible in a normal object listing, and this is the only mechanism that removes them.
The interaction with retention locks is the exam-relevant subtlety. Lifecycle rules and Object Lock both touch deletion and Object Lock wins: a lifecycle expiration cannot remove a version that is still under a retention period or a legal hold. The combination is actually the correct design for a fixed-term obligation — Object Lock in compliance mode holds the records for the required seven years, and a lifecycle rule expires them shortly afterwards, so the obligation ends on schedule instead of becoming a permanent archive. Build both halves at the same time, because a retention control with no expiry plan silently turns into indefinite retention.
The other named mechanisms follow the same shape. An EFS lifecycle policy moves files to the Infrequent Access or Archive storage class after a period without access and can transition them back on access, which is cost management with a retention flavour rather than deletion. FSx for Lustre backup policies set automatic daily backups with a retention period, plus on-demand backups, and the backups inherit the file system’s encryption. And an AWS Backup plan has its own lifecycle: when to move a recovery point to cold storage and when to expire it — bounded, where a vault lock exists, by the minimum and maximum retention the lock enforces.
One object’s regulated life, and every rule that touches it
- Day 0 — written with a seven-year compliance retention
Object Lock in compliance mode stamps a retain-until date into the version’s metadata. From this moment no principal, including root, can delete or overwrite this version, and the period can be extended but never shortened.
- Day 30 — lifecycle transitions it to a colder class
Transitions are orthogonal to retention: moving an object to a Glacier class changes where it is stored and what retrieval costs, not whether it can be deleted. Mind the minimum-duration charges on cold tiers, which can make an early transition cost more than it saves.
- Day 400 — a new version is written to the same key
In a versioned bucket the old version becomes noncurrent and both exist. Retention protects versions, not keys, so the new version carries its own retention from the bucket’s default configuration — which is why the default configuration, not the per-object call, is what makes the policy reliable.
- Any day — lifecycle expiration tries to delete it
The rule fires and fails for this version: a lifecycle expiration cannot remove an object version that is still under a retention period or a legal hold. Lifecycle does not override Object Lock, and that precedence is the whole reason the two can be combined safely.
- Day 2,556 — the retain-until date passes
The version becomes deletable for the first time. Nothing happens automatically: a retention period expiring does not delete anything, which is how “seven-year retention” quietly becomes twenty-year retention in buckets with no expiry rule.
- Day 2,586 — the expiration rule finally succeeds
The lifecycle rule removes the version a month after the obligation ends. This is the second half of the retention design: the data is gone on a schedule, which satisfies both the keep-it and the do-not-keep-it-longer requirements. Noncurrent-version expiration and delete-marker cleanup finish the job.
- Throughout — abort incomplete multipart uploads
Unrelated to the record, and it belongs in every bucket: failed uploads leave billable parts that no object listing shows. A rule aborting them after about seven days is pure housekeeping that nothing else does for you.
Designing a retention period is a short exercise that teams routinely skip, and it has four questions.
What is the obligation, exactly, and when does its clock start? Seven years from the transaction, or from the end of the financial year, or from account closure are three very different rules. The clock’s start date decides whether a fixed retain-until date works or whether you need a per-object calculation at write time.
What is the maximum you are willing to hold? Over-retention is a liability: more data to breach, more to disclose, more to pay for. If the obligation is seven years, the design target is seven years and a margin — not “indefinitely, to be safe”.
Who must not be able to shorten it, and who must be able to extend it? That answer picks the mode: compliance for an obligation nobody may override, governance where an audited team retains an emergency path. Pair either with a bucket policy setting minimum and maximum allowable retention so a mistyped date cannot create a century-long commitment.
How will you prove it? CloudTrail data events for the bucket show the deletions; S3 Inventory reports retention status across millions of objects; AWS Config rules report buckets missing lifecycle configuration or Object Lock. “We have a policy document” is not evidence; a report is.
One caution on the cold tiers, because it is a real design error rather than a trivia point: transitioning small objects to a Glacier class can cost more than leaving them in Standard, because each object carries per-object overhead and a minimum storage duration. Lifecycle rules can filter by object size for exactly this reason — transition the large objects, leave the small ones alone, and expire both on schedule.
| Mechanism | Transitions | Deletion / expiry | Interaction worth knowing |
|---|---|---|---|
Between storage classes by age, filtered by prefix, tag or object size | Current versions, noncurrent versions, expired delete markers, incomplete multipart uploads | Cannot delete a version under Object Lock retention or a legal hold — Object Lock takes precedence | |
To Infrequent Access or Archive after a period without access, and back on access | No deletion action — EFS lifecycle is about storage class, not retention | Deletion of EFS data is an application or AWS Backup concern, not a lifecycle one | |
FSx for Lustre backup policy | Not applicable — backups are a separate copy | Automatic daily backups with a retention period, plus on-demand backups | Backups inherit the file system’s encryption key; at-rest encryption cannot be turned off |
AWS Backup plan lifecycle | Recovery point to cold storage after a set period | Expire the recovery point after a set period | Bounded by the minimum and maximum retention a Vault Lock enforces — the lock outranks the plan |
Snapshot and AMI schedules, with cross-Region copy | Retention count or age per schedule | EBS-specific; AWS Backup is the broader service when you need one plan across resource types |
Key terms: S3 Lifecycle configuration, EFS lifecycle policy, S3 Object Lock, AWS Backup Vault Lock
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.