Lifecycle rules and their traps

Lesson 2 of 4 in Lifecycle and Tiering.

An S3 Lifecycle configuration is a list of rules, each with a filter and one or more actions. The filter can match a prefix, object tags, an object-size range, or nothing at all — in which case the rule applies to the whole bucket, which is a decision people make by accident. The actions divide into two families: transition an object to another storage class, and expire it.

Rules are evaluated once a day by a background process, so nothing happens the instant an object crosses a boundary, and the transition or expiration is asynchronous. That is why a day-30 rule does not act at exactly 30 days and why a bill does not drop the moment a rule is written.

Four things stop a transition, and all four have produced a support case:

The object is younger than the transition day. The obvious one, and the only one anybody checks.

The object is smaller than 128 KB. Since September 2024 the default behaviour prevents objects under 128 KB transitioning to any storage class, because a transition request is charged per object and for a small object that charge exceeds the storage it would save. You can override it with an explicit ObjectSizeGreaterThan filter, and configurations created before the change keep the old behaviour until you edit them.

The rule is invalid. Nothing may transition to Standard-IA or One Zone-IA before 30 days, so a rule that tries is rejected rather than quietly ignored.

An expiration rule reaches the object first. Lifecycle applies expiration too, so an object already past the expiry age is deleted rather than archived.

And then the charge that is not a blocker but a bill: minimum storage duration. Move an object to Glacier Deep Archive and delete it a month later, and you are billed for the remaining 150 days anyway. An aggressive tiering rule paired with a short retention rule can cost more than leaving the data in Standard the whole time — which is the arithmetic the next lab makes visible.

Transitions only move down the waterfall: Standard → Standard-IA or One Zone-IA → Glacier Instant Retrieval → Glacier Flexible Retrieval → Glacier Deep Archive, and Intelligent-Tiering sits beside the chain managing its own tiers. Nothing in a lifecycle configuration can warm an object back up. Bringing it back means copying the object over itself with a new storage class, or — for the archive classes — a restore.

The expiration actions are where versioned buckets get expensive, and where most unexplained bills live.

Expiration on a versioning-enabled bucket does not delete anything: it writes a delete marker, and the previous version becomes noncurrent and keeps being billed. NoncurrentVersionExpiration is the action that actually removes storage, optionally keeping the most recent N versions with NewerNoncurrentVersions. ExpiredObjectDeleteMarker removes delete markers that no longer have any version behind them — invisible clutter that slows listings. And AbortIncompleteMultipartUpload cleans up parts of uploads that were never completed, which are billed as storage and do not appear in an ordinary object listing at all.

That last one deserves its own sentence, because it is the answer to a recurring question shape: storage charges that do not match the objects anyone can see. A job that dies partway through a large upload leaves its parts behind; nothing cleans them up by default; ListObjectsV2 shows nothing. One rule aborting incomplete uploads after seven days is recommended on effectively every bucket.

Finally, the tools that tell you where to point a rule. S3 Storage Lens reports usage and activity across an account or organisation — average object size per bucket, the share of bytes in noncurrent versions, the bytes sitting in incomplete uploads — and makes cost recommendations. S3 Inventory delivers a flat file of every object with its size, class, encryption and replication state, which you can register as a table and query. And S3 Batch Operations takes an inventory report as its manifest and applies one action to every object in it, which is how a change reaches data that already exists — because lifecycle rules act on age and class, never on an arbitrary list.

Key terms: S3 Lifecycle configuration, Transition waterfall, Noncurrent version expiration, Incomplete multipart upload, S3 Storage Lens, S3 Batch Operations

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.