Failure, retry, and idempotency

Lesson 5 of 5 in Orchestrating the Pipeline.

A retry is not a repair. Every orchestrator will retry a failed task, which means every task will run more than once, which means the only question that matters is what the second run does.

The failure case to reason about is not a task that fails before doing anything — that one is harmless, because a re-run reads the same input and produces the same output. It is a task that fails half-done: it wrote 600,000 of its million rows and then died. That is the case idempotency exists for, and it is the case people reason about wrongly.

Four shapes of idempotent write, and the exam rewards recognising which one a scenario admits.

Deterministic output paths. Write to s3://bucket/dt=2026-10-21/part-0001.parquet and a re-run overwrites its own partial output rather than adding a second copy. This is why overwrite-by-partition beats append for anything that might be retried, and the cost you accept is that you can only write whole partitions.

Conditional writes. A DynamoDB PutItem with a condition on the item's own version, or an S3 conditional write, so the second attempt fails harmlessly instead of duplicating.

Upsert on a natural key. MERGE into a table keyed on the record's business id. Correct, and it needs a sink that can transact — an open table format or a warehouse, not plain files on S3.

Transactional commit. The sink makes the data and the metadata visible in one step, so a half-finished retry is invisible.

And the anti-pattern to recognise instantly: append plus an auto-generated surrogate key. Every retry adds rows, nothing detects it, and the error surfaces weeks later as a total that is 0.3% too high.

Three further disciplines follow, and together they are what separates a pipeline you can operate from one you nurse.

Stage and promote. Write to a staging prefix, then promote on success as a separate fast step. Two reasons, both load-bearing: readers never see a half-rewritten table, and a failed run leaves the old (wrong but consistent) data in place rather than a mixture of old and new. A dead-letter route makes a failure visible; it does not clean up the partial write, which is why staging is the structural answer.

Parameterise by period, never by the wall clock. A job that reads "yesterday" computed from now() cannot be backfilled at all; one that takes a date parameter and writes to the matching partition can be run for any day, in any order, by anyone. Airflow formalises this as the data interval, which is why backfill is a command there rather than a project.

Distinguish "the pipeline failed" from "the data is wrong". They are different questions with different detections. A job that fails is loud and a job that succeeds with bad output is silent, which is why a data-quality check is a pipeline component rather than a nicety. A row count far below yesterday's, a null rate that jumped, a sum that does not reconcile — those are the signals that catch a green run with a wrong table, and nothing in the orchestrator can produce them.

Interactive sorting exercise: Each card describes a write. Sort it by what a retry of that task would do — the point being that only one of these three outcomes is acceptable, and the middle column is where most real pipelines live.

Retry controls by service. The shapes differ, and knowing which knob exists where is what a troubleshooting question tests.
Where	Retry control	Failure destination	What it does not do
Step Functions state	`Retry` with `IntervalSeconds`, `MaxAttempts`, `BackoffRate` and jitter, per error name	`Catch` routes to a named state — so a failure path is part of the definition	make a non-idempotent task safe; it will cheerfully re-run an appending job
Airflow task	`retries` and `retry_delay`, with optional exponential backoff	an `on_failure_callback`, or a downstream task with a `trigger_rule`	clean up a partial write — the task re-runs from the top
SQS consumer	redelivery after the visibility timeout, up to `maxReceiveCount`	the dead-letter queue named in the redrive policy	help if the timeout is shorter than the processing time — then it duplicates by design
Lambda on a stream	`MaximumRetryAttempts`, `MaximumRecordAgeInSeconds`, `BisectBatchOnFunctionError`	an `OnFailure` destination receiving metadata about the batch, not the records	preserve the records — they stay in the stream until retention expires
Lambda, async invoke	two automatic retries, then the configured destination	an on-failure destination, or a function-level dead-letter queue	apply to a poll-based event source mapping, which has its own controls
Glue job	a job-level retry count	a conditional trigger on the `FAILED` state	advance or roll back a bookmark — a failed run leaves the bookmark where it was
EventBridge Scheduler	maximum retry attempts and a maximum event age per schedule	a dead-letter queue per schedule	retry the work — only the attempt to invoke the target

The sentence to remember from this module

A pipeline that finishes green is not evidence that the data is right.

In the simulation above, every task eventually succeeded, no retry budget was exhausted, no alarm fired, and the table held 2.6 times the rows it should. Nothing in Step Functions, Airflow or Glue can detect that, because from the orchestrator's point of view nothing went wrong.

Three things close the gap, and all three are design decisions made before the incident. Make every write idempotent, so a retry cannot duplicate. Stage and promote, so a partial write is never visible and a failure leaves consistent old data rather than a mixture. And run a data-quality check as a pipeline step that can fail the run — row counts against the previous period, null rates, a reconciliation sum — because that is the only component in the whole architecture whose job is to notice that a successful run produced the wrong answer.

Key terms: Idempotent write, At-least-once delivery, Exactly-once processing, Dead-letter queue, Backfill, Visibility timeout