Lineage and traceability

Lesson 3 of 5 in Maintaining and Monitoring.

Lineage is not documentation. It is the answer to two questions that only ever get asked at the worst possible moment.

"Who do I have to tell?" A column in a curated table has been wrong for three days. Downstream lineage tells you which tables, which dashboards, which extracts and which teams consumed it — so the conversation is a list rather than an apology to everybody. Without it, the honest answer to "who was affected" is "we think probably nobody important", which is not an answer.

"Where did this number come from?" A figure in a board pack is implausible. Upstream lineage walks back through every transformation to the source, so you can find the join that duplicated rows or the source system that changed a unit. Without it, you re-derive the pipeline from code while people wait.

A third use justifies the effort in calm weather: impact analysis before a change. "Can we drop this column?" is answerable in seconds with lineage and is a week of grep otherwise.

The modern mechanism is OpenLineage, an open specification for emitting lineage as events rather than drawing it as a diagram. Four pieces: a run (one execution), a job (the thing executed), datasets (its inputs and outputs), and facets — extensible metadata attached to any of them, including column-level mappings and row counts. A job emits a start event and a complete or fail event, and the graph assembles itself from the stream.

Interactive sorting exercise: Each question below is answered by one kind of traceability. Sort them — the point of the exercise is that lineage, a run id and an audit log are three different instruments and are routinely confused.

Key terms: OpenLineage, Amazon DataZone, Data lineage, Glue Data Catalog

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.