Bookmarks and incremental processing

Lesson 3 of 4 in Automating Data Processing.

Reprocessing everything, every run, is the simplest correct pipeline there is. It is also the one that costs ten times what it should by month three, which is why every real pipeline eventually has to answer: what is new since last time?

Answering it needs exactly three things, and naming them separately is what makes the failures diagnosable.

A monotonic coordinate. Something that only increases, so "newer than X" is meaningful. An object's last-modified time. A database column that never goes backwards. A partition date. If the coordinate can be reused, back-filled or issued out of order, every conclusion built on it is wrong and nothing will tell you.

A durable record of how far you got — the watermark. Written where a crash cannot lose it, and advanced after the output commits rather than before, because advancing it first converts a crash into permanent data loss.

An idempotent write. Because you will replay. Retries replay, backfills replay, and at-least-once delivery replays whether you planned to or not.

AWS Glue job bookmarks are a managed implementation of all three — and the thing to understand before using them is that the managed-ness hides the coordinate. When a bookmark goes wrong, the job succeeds.

So, precisely: what does a bookmark store?

Not one number for the job. A map from source operator to state, keyed by that operator's transformation_ctx — which is why the string matters and why renaming it orphans the state. Inside each entry, the content depends on the source:

  • Amazon S3. The last-modified times of objects already processed. The next run reads objects whose last-modified time is newer. This has a consequence people meet in production rather than in documentation: an object overwritten with the same key gets a new last-modified time, so Glue reads it again — and if the pipeline appends, that data is now in the target twice.
  • JDBC. The high value of one or more bookmark keys, defaulting to the table's primary key, combining into a single compound key. The next run reads rows above that value.

And the three settings, which are a closed set worth memorising because a scenario will name one: Enable tracks and advances the state; Disable — the default — processes everything every run; Pause reads the state and processes from it but does not advance it, which is how you re-run the same window deliberately, and which is also what somebody left switched on when a pipeline mysteriously reprocesses the same four hours for a week.

The six ways a bookmarked job silently reprocesses

All six produce the same symptom — duplicate rows downstream, with every job run reporting success.

  1. job.init() or job.commit() missing. init loads the state; commit persists it. Without the pair there is nothing to read and nothing to write, so every run starts from zero.
  2. transformation_ctx missing on the source read. That operator is simply not tracked, so it reads the whole source every time. Occasionally deliberate — a small dimension re-read in full next to an incremental fact feed — but usually an omission.
  3. transformation_ctx duplicated across two operators. Two sources sharing one key overwrite each other's state, and both end up wrong.
  4. Spark DataFrames instead of DynamicFrames. Bookmarks are a Glue DynamicFrame feature; a plain Spark spark.read is invisible to them.
  5. Concurrent runs of the same bookmarked job with MaxConcurrentRuns above 1. Two runs read the same state and both advance it; the overlap is processed twice.
  6. The source objects were modified. Rewriting an object in place gives it a new last-modified time, and Glue is correct to re-read it — the pipeline’s append is what makes it a duplicate.

The two ways it silently skips — which is worse

Duplicates are loud if you ever count anything. Skips are not.

A non-monotonic JDBC bookmark key. The stored high value is 10,000. The upstream system issues id 9,998 for a corrected record. Every run from then on reads rows above 10,000, so 9,998 is never read again — and nothing fails, nothing warns, no count is obviously wrong. The fix is to bookmark on a column the database guarantees increases, such as a server-side updated_at, not on a business identifier an upstream system controls.

Updates to already-processed rows. A bookmark on an insert-time column is blind to a row that was changed after it was read. If the source mutates history, a bookmark is structurally the wrong mechanism and change data capture is the right one.

Test for both the same way: run the pipeline, then reconcile a count and a sum against the source for a closed window. A DQDL RowCountMatch or AggregateMatch rule turns that reconciliation into something that runs every night rather than something you do once after an incident.

When to write your own watermark instead

A bookmark is the lowest-effort option and the least visible. Write your own watermark when you need any of these:

  • To read, audit or correct it. A DynamoDB item or a Glue workflow run property can be inspected and edited; bookmark state cannot be read as a value.
  • To advance it transactionally with your own commit. Then a crash between write and watermark-advance costs you a replay, not a gap.
  • To backfill by setting the coordinate to a past value — the operation a job that reads "today" from the system clock cannot perform at all.
  • To use a coordinate Glue does not model, such as a logical batch id or a partition date supplied by an upstream contract.

The cost is a few lines of code and one more thing to monitor. The benefit is that the hardest question in this lesson — how far did we actually get? — becomes a value you can SELECT.

The alternatives, and when each one wins

Bookmarks are not the only way to process only what is new.

Partition-based processing. Pass the partition date as a job parameter and read exactly that partition. No state at all: the parameter is the coordinate, the job is trivially re-runnable for any past date, and the orchestrator holds the schedule. This is often the better design, and it is why a Distributed Map over a list of dates is such a common backfill shape.

Open table formats. Iceberg, Hudi and Delta Lake track snapshots, so "everything since snapshot N" is a query the table itself answers — and MERGE makes the write idempotent by business key rather than by partition.

Change data capture. When the source mutates rows, nothing watermark-shaped is correct. Read the database's change log instead.

The exam cue is usually explicit: files only ever appended points at bookmarks or partitions, records are updated in place points at change data capture or a MERGE.

Key terms: Glue job bookmark, transformation_ctx, Bookmark key, Watermark, Idempotent pipeline step

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.