S3 as the data lake
Lesson 2 of 5 in Choosing a Data Store.
Amazon S3 is the default answer to "where does the data live" on AWS, and almost every misunderstanding about it comes from one fact: there are no directories. An object key is a single flat string. The slashes in events/dt=2026-03-14/part-0.parquet are ordinary characters that consoles and query engines choose to render as a tree, and nothing in the service treats them as structure.
Three consequences follow, and all three show up as exam questions.
Listing is a scan of a sorted key space. ListObjectsV2 with a prefix is cheap because the keys are stored in order and the prefix bounds the range. Listing a bucket to find one object is not cheap, and a job that begins by enumerating a million keys has already spent more time than the read will take. This is also why the partition-discovery tools behave the way they do: MSCK REPAIR TABLE lists the whole prefix every time it runs.
Request rate scales per prefix, not per bucket. Amazon S3 supports at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix, and there is no limit on the number of prefixes. So a job returning 503 Slow Down is usually a job whose keys all share one prefix; spreading the key space fixes it, and S3 also repartitions automatically as sustained rates rise, which is why the throttling is often temporary. You no longer need to randomise key names to get this — sequential, logical prefixes are fine — but you do need several of them.
There is nothing between you and a bad write. S3 gives strong read-after-write consistency for every object operation, so a reader will never see a stale version of an object. What it does not give is any validation that the object matches the schema the catalog claims, which is the whole of schema-on-read and the reason a lake needs hygiene rules that a database would enforce for you.
A horizontal bar chart of gigabytes scanned to answer one query — select two columns for one day out of a month of event data — under five storage layouts. Uncompressed CSV with no partitioning scans 400 gigabytes, the whole month. Gzip-compressed CSV with no partitioning scans 80 gigabytes, because compression reduces the bytes but every row and column is still read. Uncompressed Parquet with no partitioning scans 20 gigabytes, because only the two named columns are read. Snappy Parquet partitioned by day scans 0.65 gigabytes, because only one day of two named columns is read. Snappy Parquet partitioned by day with a filter that also matches the file-level column statistics scans 0.18 gigabytes, because row groups whose minimum and maximum values cannot satisfy the predicate are skipped without decompression. The teaching point is that the first three steps each remove roughly an order of magnitude and none of them required changing the query.
| Format | Layout | Carries its schema | Splittable | Choose it when |
|---|---|---|---|---|
CSV | row, text | no — the header is a convention, and may not be there | yes, unless gzipped | something else produced it and you have no say; land it in raw and convert |
JSON (lines) | row, text | partly — field names travel, types do not | yes, one record per line | records are irregular or nested and you are still in the raw zone |
row, binary | yes — the writer schema travels with the data or by id | yes, block-level | streaming transport and schema-registry workflows, and Hudi’s delta logs | |
columnar, binary | yes, in the footer | yes, by row group | the default for anything anyone will query — read by name in Athena | |
columnar, binary | yes, in the stripe footers | yes, by stripe | an existing Hive estate expects it — read by index in Athena, which changes what schema changes are safe |
Two newer pieces of Amazon S3 change store decisions and are fair game in a current exam.
Amazon S3 Tables introduces a third kind of bucket — a table bucket, addressed through the s3tables API, holding namespaces and tables rather than free-form keys — that stores Apache Iceberg tables and runs their maintenance for you. That matters because a self-managed Iceberg table needs three recurring jobs nobody wants to own: compaction, snapshot expiry, and removal of files no snapshot references any more. S3 Tables does all three with configurable defaults — target file size 512 MB, at least one snapshot retained, snapshots expiring at 120 hours, unreferenced files marked after 3 days and deleted after 10 more. Note what the second default implies: a 120-hour snapshot window is also your time-travel window, and those deletions are permanent.
S3 Express One Zone is a storage class inside a directory bucket, co-locating data with your compute in a single Availability Zone to cut request latency by an order of magnitude. It is the right store for data a query engine reads thousands of times — a hot working set, shuffle spill — and the wrong one for a landing zone, for three reasons a stem will lean on: it is one Availability Zone, so the data needs a durable copy elsewhere; storage is priced well above S3 Standard while requests are priced well below, so it only pays on genuinely hot data; and directory buckets do not support every general-purpose-bucket feature, so it is not a drop-in replacement.
Key terms: S3 prefix, Columnar format, Small-file problem, Amazon S3 Tables, Data lake
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.