Git flow and CI/CD for data

Lesson 3 of 4 in Programming Concepts on the Exam.

A data pipeline has two artefacts that must move together: the code and the infrastructure definition. That is the whole reason a data engineer is asked about Git strategy, and it is what makes long-lived branches worse here than elsewhere — a branch of infrastructure code is a stack that has drifted from production in ways nobody has tested.

Trunk-based development keeps main always releasable, merges small changes continuously, and hides unfinished work behind a feature flag rather than behind a branch. GitFlow is the named counter-example, with long-lived develop, release and hotfix branches; it suits versioned products shipped to customers far better than a continuously deployed pipeline.

The pattern that goes with it is environment promotion: the same template or CDK app deployed to dev, then staging, then production, with parameters as the only difference. That is what makes "it worked in staging" mean something. If the environments are built from different templates, it means nothing at all.

And here is the data-specific wrinkle that a stem will test. A feature flag can hide a code path. It cannot hide a schema change. Those need the expand-and-contract discipline instead:

  1. Expand — add the new column, nullable, and deploy. Old readers ignore it.
  2. Dual-write — populate both the old and new columns, and deploy. Old readers still work.
  3. Migrate readers — move consumers to the new column, one at a time, and deploy each.
  4. Contract — stop writing the old column, and only then drop it.

Three or four deployments, never one. Attempting it in a single release means a window in which some readers are looking for a column that no longer exists, and in a lake that window is as long as the slowest consumer takes to redeploy.

Expand and contract — the only safe way to change a schema readers depend on

  1. A column must be renamed or retyped

    Several consumers read this table and redeploy on their own schedules. A single release cannot change them all at once.

  2. 1 · Expand — add the new column

    Nullable, alongside the old one. Deploy. Every existing reader ignores a column it does not know about, so this step is safe by construction.

  3. 2 · Dual-write both columns

    The producer populates old and new. Deploy. Old readers still work; new readers can start. This is the step a feature flag genuinely does cover, because it is a code path.

  4. 3 · Backfill the new column for history

    A parameterised job over past partitions, with an idempotent write. Until this completes, the new column is correct only for recent data — which means a new reader must not be switched over yet.

  5. 4 · Migrate readers, one at a time

    Each consumer redeploys to use the new column on its own schedule. This is the long step, and its duration is set by the slowest team rather than by you.

  6. All readers confirmed moved?

    Verify rather than assume — query logs, Lake Formation access records, or simply ask. A reader you forgot is the one that breaks in step 5.

  7. 5 · Contract — stop writing the old column

    Stop populating it, deploy, and wait. Leaving it in place but stale is a deliberate soak period: if something breaks, nothing has been dropped yet.

  8. 6 · Drop it

    Only now. The column is gone, the migration is complete, and the whole sequence took four to six deployments — which is the honest cost of changing a schema that other people read.

  9. A reader is still on the old column

    Stop and wait. The old column is still being written, so nothing is broken and there is no urgency. Dropping it now is the only irreversible step in the sequence — which is exactly why it is last.

Key terms: Trunk-based development, Change set, Backfill, Idempotent write