Partitioning, lifecycle, and storage cost
Lesson 3 of 4 in Data Architecture for ML: Lakes, Warehouses, Catalogs.
A partition is a column encoded into the storage path — s3://lake/events/dt=2026-10-01/region=eu/… — and its one rule of design: partition by what reads filter on. Training jobs pull date ranges, so date is nearly always the first key; region or source earn their place only when queries actually filter by them. Get it right and engines prune everything outside the filter; get it wrong — partition by a column nobody filters on, or by one with millions of values — and you trade scan savings for catalog bloat and a lake of tiny directories.
Granularity has a floor because files have overhead: every object costs a listing entry, a request, and a task-scheduling decision. Aim for files in the hundreds of megabytes; a training job that must list two million kilobyte-sized objects spends its opening minutes on S3 LIST calls before a single byte of learning.
Horizontal bar chart of gigabytes scanned by a query filtering for a single day over a hypothetical 365 GB year-long table. With no partitions the engine scans all 365 GB. Partitioned by date it scans about 1 GB — only that day's partition. Partitioned and stored as Parquet it scans about 0.1 GB, because only the referenced columns of the one partition are read.
| Class | Access pattern | Retrieval behavior | Training implication |
|---|---|---|---|
Standard | Hot — active training and feature pipelines | Immediate | The default for datasets in current use |
Intelligent-Tiering | Unknown or shifting | Immediate; objects move tiers automatically by observed access | Set-and-forget for mixed-age lakes |
Standard-IA | Known-infrequent, still needed on occasion | Immediate, with a per-GB retrieval charge | Older dataset versions you occasionally retrain from |
Glacier Instant Retrieval | Rare (think quarterly) | Still immediate, higher retrieval charge | Archives that must stay directly readable |
Glacier Flexible / Deep Archive | Archive and compliance | Restore required before any read — minutes to many hours (Deep Archive the slowest) | A training job pointed here fails until objects are restored — never lifecycle active training data into these |
Lifecycle rules automate the march: keep the active training window in Standard, transition partitions past a chosen age to IA or Glacier Instant Retrieval, expire scratch and stale object versions. The exam's favorite mismatch is pairing a deep-archive tier with a live workload — "monthly retraining" plus "Deep Archive" in the same stem is the tell: either the lifecycle policy is wrong, or a restore step is missing.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.