Event-driven triggers

Lesson 4 of 5 in Orchestrating the Pipeline.

A scheduled pipeline has to guess when its input will be ready. An event-driven one does not, and that is the whole argument: the arrival is the trigger, so there is no window to miss and no empty run to mistake for a successful one.

S3 event notifications are configured per bucket with an event type (s3:ObjectCreated:*, s3:ObjectRemoved:*, and others) and optional prefix and suffix filters. Delivery is at-least-once, which means a duplicate notification for one object is normal and the handler must be idempotent.

There are two routes out of the bucket and the difference matters.

Direct notifications to Amazon SNS, Amazon SQS or AWS Lambda are the classic path: simple, low latency, and limited — the filters are prefix and suffix only.

EventBridge integration, enabled with one setting on the bucket, publishes every S3 event to the default bus instead. Rules can then match on any field of the event (size, key pattern, requester), fan one event out to several targets, be archived and replayed, and be routed cross-account. For anything beyond "call this one function", this is the better answer, and the archive-and-replay capability is genuinely useful during an incident.

And one pattern worth recognising on sight for a high-volume landing zone: S3 → SQS → a consumer. The queue absorbs bursts, gives you a retry and a dead-letter queue for free, and decouples the arrival rate from the processing rate. S3 → Lambda directly will happily attempt ten thousand concurrent invocations and meet your account concurrency limit instead.

Which trigger mechanism the requirement wants. The giveaway is usually in the phrasing of *when* something should happen.
Mechanism	Fires on	Brings	The phrase that points at it
S3 → EventBridge	an object created, removed or restored	rules on any event field, fan-out, archive and replay, cross-account routing	"when the file arrives", "as soon as it lands"
S3 → SQS → consumer	the same events, buffered	burst absorption, batching, a retry and a dead-letter queue, decoupled rates	"thousands of files at once", "the function keeps getting throttled"
EventBridge Scheduler	a cron or rate expression, or a one-time timestamp	time zones with daylight-saving handling, flexible time windows, per-schedule retries and DLQ	"every night at 02:00 local time", "the job ran an hour early after the clocks changed"
An EventBridge rule on a service event	a Glue job state change, a DMS event, a Config rule going non-compliant	reaction to AWS-side state without polling	"when the crawler finishes", "if the job fails, notify"
EventBridge Pipes	a record on a stream, queue or change feed	filtering before the target is billed, plus optional enrichment	"move the change feed into Firehose, but only the rows we care about"
An Airflow sensor	a condition becoming true, checked on an interval	natural fan-in — wait for several inputs before anything starts	"all three regional extracts must be present before we aggregate"
A Glue conditional trigger	a listed set of Glue jobs and crawlers reaching listed states	fan-in within an all-Glue pipeline, with `AND` or `ANY`	"after both crawlers succeed, run the aggregation job"

The clock-change bug, and why Scheduler exists

An EventBridge rule with a cron expression is evaluated in UTC. So a job intended for 02:00 local time runs at 02:00 local in winter and 03:00 local in summer, or the reverse — silently, twice a year, and only in the regions that observe daylight saving.

For a data pipeline that matters more than it sounds. "Process yesterday" computed from the wall clock at the wrong local hour can process the wrong day entirely, and a nightly window that was comfortably after the upstream extract becomes an hour before it.

EventBridge Scheduler supports a time zone per schedule with automatic daylight-saving adjustment, which is the direct fix. It also brings three things a rule does not: one-time schedules at a timestamp, a flexible time window so that thousands of schedules do not all fire on the same second, and a retry policy with a dead-letter queue per schedule.

If a stem mentions a job that ran at the wrong local time after a clock change, or thousands of schedules overwhelming a downstream service at midnight, it is pointing at Scheduler.

Key terms: S3 event notification, EventBridge Scheduler, EventBridge Pipes, Dead-letter queue, Amazon EventBridge