Choosing an orchestrator

Lesson 1 of 5 in Orchestrating the Pipeline.

Every orchestrator on this exam can run job A, wait, then run job B. Comparing them on that is useless. What differs is five things, and a stem will state one of them precisely while leaving the rest vague — that one is the answer.

How a failure is expressed. Step Functions has Retry and Catch as first-class fields on a state, so a failure path is part of the definition rather than something you remember to write. Apache Airflow has task-level retries, callbacks and trigger_rule, which is more expressive and lives in Python you maintain. Glue workflows have conditional triggers on job state and nothing finer.

Dependency complexity. A linear chain suits anything. Fan-in — a task waiting on several upstreams with AND or ANY semantics — is where tools separate, and branching on a computed value rather than on a job's state is where Glue workflows stop.

Who maintains the graph. A DAG in Python is enormously flexible and is code: a team must own it, test it, and upgrade Airflow under it. A state machine in JSON or an infrastructure-as-code construct is less expressive and much harder to break.

Cost at idle. This is the axis people forget and it decides more real choices than any other. MWAA bills for the environment while it exists, not per execution, so an idle environment has a floor. Step Functions Standard bills per state transition, so a nightly twelve-step pipeline costs almost nothing. For a small pipeline that difference dominates everything else.

Whether a human is in the loop. An approval step needs the callback pattern, which is a Standard-workflow feature — and noticing that it is incompatible with Express is sometimes the whole question.

The four orchestrators on the axes that decide between them. Read the row the stem states most precisely.
	Step Functions	MWAA (Airflow)	Glue workflows	EventBridge
The graph is	a state machine in JSON or an IaC construct	Python code you write and own	jobs and crawlers wired by triggers, in the console or API	not a graph at all — a router and a scheduler
Failure semantics	`Retry` and `Catch` per state, with backoff and jitter fields	task retries, callbacks, `trigger_rule` — expressive, and yours to maintain	conditional triggers on job state only, with `AND` or `ANY`	a retry policy and a dead-letter queue per target
Branch on a computed value	yes — `Choice` states	yes — anything Python can express	no — only on job and crawler states	only by event pattern on the event itself
Cost shape	per state transition (Standard) or per request (Express) — nothing at idle	per environment-hour while it exists, regardless of executions	no orchestration charge — you pay for the jobs	per million events; effectively nothing at this scale
Human approval	yes — the task token pattern, Standard only	yes, with a sensor or a custom operator	no	no
Massive fan-out	Distributed Map — up to 10,000 parallel child executions	dynamic task mapping, bounded by worker capacity	no	fan-out to targets, not iteration over a dataset
Reach for it when	AWS services to sequence with real branching and retries, and nothing at idle	existing Airflow DAGs, many third-party systems, or backfill as a routine operation	the pipeline is entirely Glue jobs and crawlers	something should start when an event happens, or on a schedule

Which orchestrator does this pipeline need?

Interactive decision tree — outcomes:

A Glue workflow
For an all-Glue pipeline this is the lowest-overhead option available: nothing to provision, no separate IAM surface, a run graph in the console, and run properties to pass values between nodes. Fan-in is expressed with a conditional trigger listing several jobs and AND semantics.

The moment the pipeline must call Lambda, run an EMR step, wait for a human or branch on a computed value, it has outgrown this and belongs in Step Functions.
AWS Step Functions
The default for sequencing AWS services. Retry and Catch are fields on a state rather than code you remember to write; Choice states branch on values; optimised integrations start a Glue job or an EMR step and wait for it without a polling loop; and Distributed Map handles "process every object under this prefix" as one state.

Use Standard unless the executions are very short and very numerous. Nothing is billed at idle.
Step Functions with the callback pattern
.waitForTaskToken passes a token to the target and pauses the state — not polling, not billed for waiting — until something calls SendTaskSuccess or SendTaskFailure. That is the clean way to do a manual approval before a destructive backfill, or to wait on an on-premises system.

Note the constraint, because it is a frequent discriminator: callbacks are Standard only. An Express workflow cannot wait on a task token and could not usefully anyway, since its ceiling is five minutes.
Amazon MWAA
Choose it for one of three honest reasons: the team already has Airflow DAGs; the pipeline touches many third-party systems the provider ecosystem already covers; or the graph itself needs real Python — shared helpers, dynamic structure, non-trivial logic.

What it costs: the environment is billed while it exists, so there is a floor an idle Step Functions state machine does not have, and you own Airflow's operational surface — version upgrades, worker sizing, a scheduler that can fall behind, and a metadata database that grows until somebody prunes it.
An S3 event notification, via EventBridge
Turn on EventBridge integration for the bucket rather than configuring a direct notification. Direct notifications filter on prefix and suffix only; through EventBridge you can match any field of the event, fan one event out to several targets, archive and replay, and route cross-account.

For a high-volume landing zone, insert a queue: S3 → SQS → consumer absorbs bursts, gives you a retry and a dead-letter queue for free, and decouples arrival rate from processing rate. Delivery is at-least-once, so the handler must be idempotent.
EventBridge Scheduler
Not an EventBridge rule with a cron expression — the dedicated service. It brings time zones with automatic daylight-saving adjustment (a cron rule is UTC and silently shifts an hour twice a year relative to local business time), one-time schedules at a timestamp, a flexible time window to spread load rather than firing everything on the minute, and a retry policy with a dead-letter queue per schedule.
EventBridge Pipes
A source, an optional filter, an optional enrichment and a target — which is exactly the Lambda function whose whole body would have been "receive from the queue, drop what I do not care about, reshape the rest, call the next service".

The crucial difference from that function: a pipe filters before the target is engaged, so you are not billed for the records you discard.

A DAG describes dependency, not schedule — and conflating them is a classic defect

A directed acyclic graph says "B needs A". It says nothing about when. The same graph can be triggered hourly, on an event, or by hand.

The failure that follows from conflating them is specific and common: a pipeline scheduled at 02:00 because the upstream file "usually arrives by 01:30". One night it arrives at 02:15, the job runs over an empty prefix, writes nothing, and succeeds. Nothing alarms, because no job failed.

Two correct shapes. Push: the file arriving is the trigger — an S3 event notification starts the pipeline, so there is no window to miss. Pull: the pipeline is scheduled and its first task is an sensor that waits for the input, with a timeout that alarms if it never appears.

The push has lower latency and no polling cost. The pull is simpler when several inputs must all be present before anything may start, which is exactly the fan-in case sensors are good at. What is not acceptable is a schedule with neither.

Key terms: DAG (directed acyclic graph), AWS Step Functions, Amazon MWAA, Glue workflow and triggers, EventBridge Scheduler, EventBridge Pipes