Iceberg, Hudi, and Delta Lake

Lesson 3 of 5 in Data Modeling and Schema Evolution.

A Hive-style table is a directory and a promise: everything under this prefix is the table. That is enough for append-only analytics and it is not enough for anything else, because it gives you no way to change a row, no atomic commit across several files, and no consistent read while a writer is working.

The open table formats fix all three by tracking individual data files in metadata instead of inferring a table from a listing. Apache Iceberg is the one to know in detail on AWS, and its three layers are worth being able to name.

The catalog — on AWS usually the Glue Data Catalog — holds a pointer to the current metadata file, and nothing else. The metadata layer holds the metadata file itself (schema, partition spec, table properties, and the list of snapshots), plus manifest lists indexing the manifests, plus manifest files recording every data file with its partition values and per-column minimum and maximum statistics. The data layer is ordinary Parquet, ORC or Avro.

Four properties follow mechanically, and each answers an exam question:

Atomic commits. A write adds files, then atomically swaps the catalog's metadata pointer. A reader resolves the current snapshot when it loads the table and sees a consistent file set for its whole run, so there is no half-written state to observe. File-level conflict resolution and optimistic commit retries give genuine concurrency rather than a table lock.

Time travel and rollback. Earlier snapshots still list the files they referenced, so reading the table as it was is free until those snapshots expire. That is how a report is reproduced exactly, how a suspect load is diffed, and how a bad merge is rolled back without restoring a backup.

Statistics-based pruning. Because files are listed with their min/max per column, the planner skips files on statistics rather than on directory names — so a filter on a non-partition column can still prune.

Partition evolution and hidden partitioning. Each file records its own partition values, so the table can start writing with a new partition spec while old files keep theirs, with no rewrite. And the table can derive the partition value from a column with a transform such as days(event_time), so a query filtering on event_time prunes without anybody writing a separate partition predicate.

Copy-on-write and merge-on-read is the one trade-off to be able to reason about out loud, because it is a question shape rather than a fact.

Copy-on-write rewrites the whole data file containing the affected rows. Write amplification is high and proportional to the files touched; reads stay exactly as fast as plain columnar files, because the reader merges nothing. It is the right mode for read-heavy tables whose data changes rarely — a dimension refreshed nightly, a curated mart — and the wrong mode for a continuous stream of small updates, where a 256 MB file is rewritten every few seconds for a handful of rows.

Merge-on-read appends a small delta or delete file and leaves the base file alone. The write is trivial, and every subsequent read must apply those deltas — so compaction is mandatory maintenance, not an optimisation. Without it the delta count grows until reads crawl. In Apache Hudi the table type is chosen at creation and cannot be changed afterwards, which makes it a decision rather than a setting.

The three formats differ in route rather than destination. Iceberg tracks files in manifests under an atomically swapped metadata pointer. Hudi adds an index from record key to file, so an upsert finds exactly which files to rewrite instead of joining the incoming batch against the whole table, and an incremental query returns only the records changed between two commit times — which makes a downstream job genuinely incremental rather than a full recompute with a filter. Delta Lake keeps an ordered transaction log of JSON commits with periodic Parquet checkpoints, reaching the same guarantees by replaying a log.

All three are supported by AWS Glue, Amazon EMR, Amazon Athena and Amazon Redshift. For a greenfield AWS lake the ecosystem gravity is with Iceberg — it is what Amazon S3 Tables manages — and the honest reason to choose Delta is usually that an existing Spark or Databricks estate already writes it.

Key terms: Apache Iceberg, Snapshot (table format), Copy-on-write (CoW), Merge-on-read (MoR), Apache Hudi, Delta Lake, Compaction, Partition evolution

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.