Evolving a schema safely

Lesson 4 of 5 in Data Modeling and Schema Evolution.

Schema evolution is not a technical question first. The technical question is can I make this change; the engineering question is who reads this, and which of four changes is it?

Adding an optional column is almost always safe: old readers ignore it, new readers see null for old data. Removing a column breaks any reader that still selects it. Renaming is a remove plus an add, unless the format resolves by name and the engine is told about the rename. Changing a type is safe only in the widening direction — int to bigint, float to double — because narrowing loses data and reinterpreting (string to date) fails on every value that does not parse.

In a lake, nothing enforces any of this, which is the whole of schema-on-read: a producer writes whatever it likes and the mismatch surfaces at read time, hours later, with an error naming the reader. Three countermeasures sit upstream of that, and they are not alternatives.

The AWS Glue Schema Registry moves enforcement to write time. A producer registers its schema and serialises with a schema id instead of the whole schema — so messages get smaller — and the registry validates each new version against a compatibility mode before accepting it. A producer that drops a required field under BACKWARD fails at registration rather than breaking consumers overnight. It supports Avro, JSON Schema and Protobuf, and integrates with Apache Kafka and Amazon MSK, Amazon Kinesis Data Streams, Apache Flink and AWS Lambda. Its limit is scope: it governs the streams it is wired into, not files some other job drops in a bucket.

A table format enforces its own schema and makes most changes metadata-only — add, drop, rename, reorder and widen a type without rewriting data.

A data-quality check between landing and curated catches what neither covers, which is the file that arrived from a partner with no registry and no contract.

Compatibility modes — decide from who upgrades first
Mode	Guarantees	Permits	Forbids	Use when
BACKWARD (recommended)	a consumer on the new schema can read data written with the previous one	adding an optional field	dropping a field, or changing its type	consumers are upgraded before producers — the usual order
BACKWARD_ALL	the same, against every previous version	deleting fields, adding optional fields	anything incompatible with any earlier version	old consumers may be arbitrarily far behind
FORWARD	a consumer on the old schema can read data written with the next one	adding fields, deleting optional fields	changes an old reader could not tolerate	producers are upgraded first and consumers follow
FULL	both directions, against the last version	adding or removing optional fields	everything else	neither side can be ordered
NONE / DISABLED	nothing / no new versions at all	`NONE`: any change, unchecked	`DISABLED`: versioning itself	development, or freezing a schema deliberately

One more mechanism decides what is safe, and it is a property of the file format rather than of anybody's policy: how a reader matches a file's columns to the table definition.

Amazon Athena reads Parquet by name (parquet.column.index.access defaults to false) and ORC by index (orc.column.index.access defaults to true). The consequences are exactly opposite:

In Parquet, reordering columns or inserting one in the middle is harmless, because the reader looks them up by name. Renaming a column makes its data vanish into nulls, because the name it is looking for is no longer there.

In ORC, renaming is harmless, because position is what matters. Reordering or inserting a column silently serves the wrong column's values under the right heading — the worst class of failure there is, because nothing errors and the numbers look plausible.

Either default can be overridden by a CREATE TABLE carrying the SerDe property. And here is the detail that produces a half-fixed table: the SerDe properties are not automatically propagated to existing partitions. A change at table level reaches new partitions only; the old ones have to be re-added with ALTER TABLE ADD PARTITION, which on a large table means scripting it.

Which is the strongest practical argument for a table format in this lesson: Iceberg assigns every column a stable field id and resolves by that, so add, drop, rename, reorder and widen are all metadata-only operations that cannot be confused with each other. The whole by-name-or-by-index question simply stops existing.

Interactive sorting exercise: A schema change is proposed against an existing table. Sort each one by what it will actually do — and notice that two of the answers depend on the file format rather than on the change.

The schema change that cannot be undone by rolling back the code

A deployment that writes data with a new shape is not reversible by reverting the deployment: the files are already there. Rolling back the producer stops new bad data and leaves the existing bad data in place, so every reader keeps failing until somebody rewrites or quarantines it.

That asymmetry is why the controls in this lesson all sit before the write. A compatibility mode refuses the schema version; a table format rejects or safely absorbs the change; a quality check quarantines the batch into a reject prefix instead of the curated zone. Each of them turns an unrecoverable data event into a recoverable deployment event.

And the corollary for a table you own: treat the table definition as a published interface. Four engines and an unknown number of downstream consumers resolve it by name, so the review question is not "does this work" but "who else reads this, and have they been told". A type widened on purpose with a note in the changelog is engineering; the same widening discovered by a dashboard is an incident.

Key terms: Schema evolution, AWS Glue Schema Registry, Compatibility mode, Column resolution (by name or by index), Apache ORC

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.