Schedules, events, and triggers
Lesson 1 of 4 in Automating Data Processing.
A pipeline that needs a human to start it is not a pipeline, it is a procedure. Automating it means answering one question — what makes this run? — and there are only three honest answers.
A clock. It is 02:00, so run. Simple, predictable, and built on an assumption: that whatever this job reads has already arrived. The assumption is invisible until the day the upstream export is forty minutes late, at which point the job runs on time, reads yesterday's file, succeeds, and publishes a number nobody notices is stale.
An event. The file landed, so run. The trigger is now the arrival itself, so lateness becomes latency instead of corruption. The cost is that arrival is per-object: a vendor who uploads one file becomes one run, and a vendor who uploads nine hundred becomes nine hundred unless something fans them in.
A dependency. The upstream job succeeded, so run. This is the one people under-use, and it is the only one of the three that encodes correctness rather than timing — the downstream aggregate cannot be right until the load that feeds it has finished, and saying so in a trigger is better than saying so in a comment and hoping the schedules stay forty minutes apart.
Most real pipelines use all three, and a good design states which one each stage uses and why. The exam tests the seam between them: a scenario that mentions unpredictable arrival times is not answered by a tighter cron expression.
Now the choice of orchestrator, which is a different question from the choice of trigger and is asked constantly.
Four options, and each one is eliminated by a specific sentence in a scenario. Glue triggers and workflows handle a fixed chain of Glue jobs and crawlers with no loops, no branching on a value, and no approvals; they are free, they need nothing else switched on, and they lose the moment a requirement includes any of those three words. Step Functions is the AWS-native workflow engine: retries with backoff per state, catch and branch, direct SDK integrations to hundreds of services so most steps need no Lambda at all, wait states, human approval through a task token, and Distributed Map for a fan-out over millions of objects. MWAA is managed Airflow, and the cue is almost always existing DAGs or a team already fluent in Airflow — plus two capabilities Step Functions does not match: sensors that block on an external condition, and first-class date-range backfill. EventBridge is not an orchestrator at all; it is the wiring that starts one.
The honest comparison is about what each one costs you when it is wrong. A Glue workflow that outgrows its shape has to be rewritten. An MWAA environment runs — and bills — whether a DAG does or not, and you own its dependency upgrades. Step Functions charges per state transition, which makes a chatty polling loop surprisingly expensive. None of those is a reason to avoid any of them; they are the sentences a scenario uses to tell you which one it means.
| Option | Shape it expresses | Costs when idle | Named by a scenario that says |
|---|---|---|---|
a fixed chain of Glue jobs and crawlers, ANY/ALL on completion states | nothing | "several Glue jobs in order", "least additional service to operate" | |
retries, catch, branch, wait, approval, map over millions of items | nothing — billed per state transition | "retry with backoff", "branch on the result", "fan out per partition" | |
Python DAGs, sensors that block on a condition, date-range backfill | the environment runs continuously | "existing Airflow DAGs", "our team already uses Airflow", "backfill a date range" | |
not a workflow — the clock and the event wiring that start one | nothing | "when a file arrives", "every hour in local time", "millions of schedules" |
Interactive sorting exercise: Each requirement below is decided by one of the three starting mechanisms. Sort them — the point of the exercise is that "run it more often" is never the answer to an arrival-time problem.
Key terms: EventBridge Scheduler, AWS Glue trigger, Glue workflow and triggers, Amazon MWAA, S3 event notification
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.