Step Functions for data workflows

Lesson 2 of 5 in Orchestrating the Pipeline.

Three Step Functions features turn it from a generic workflow engine into a data-pipeline tool, and each one replaces a thing people otherwise hand-write badly.

Optimised service integrations that wait. glue:startJobRun.sync starts a Glue job and the state does not complete until the job does. No polling loop, no Lambda function whose whole body is while not done: sleep(30), and no state transition billed every thirty seconds for four hours. The same pattern exists for EMR steps, Athena queries, Batch jobs and SageMaker.

Distributed Map. The Map state has two modes and the gap between them is large. Inline runs iterations inside the parent execution: maximum 40 concurrent, input must be a JSON array, and every iteration's events are appended to the parent's history. Distributed runs each iteration as its own child execution with its own history, allowing up to 10,000 parallel child executions and taking an S3 object listing, a CSV file, a JSON file or an Athena manifest as its item source.

AWS states the switch conditions plainly, and they are the exam's decision rule: use Distributed mode when the dataset exceeds 256 KiB, when the execution history would exceed 25,000 entries, or when you need more than 40 concurrent iterations. All three are the same underlying limit — the parent execution being asked to carry everything.

The callback pattern. .waitForTaskToken passes a token to the target and pauses the state until SendTaskSuccess or SendTaskFailure arrives. The state is not polling and is not billed for waiting, which makes it the right way to handle a manual approval or a system you cannot poll. It is Standard only.

Standard against Express. The cost reasoning inverts with volume, which is why the right answer depends on a number in the stem rather than on a preference.
	Standard	Express
Maximum duration	1 year	5 minutes
Execution semantics	exactly-once	at-least-once
Execution history	retained and visible in the console	not retained — logs go to CloudWatch Logs, which you must enable
Priced by	state transitions	requests and duration
Callback pattern	yes	no
Cheap when	a few long workflows — a nightly twelve-step pipeline is a handful of transitions	millions of very short executions, where per-transition pricing would be ruinous
The discriminator in a stem	"the workflow runs for two hours", "a human approves step three", "it must not run twice"	"thousands of executions per second", "per-event processing"

Two Step Functions anti-patterns that a stem will offer you

The polling Wait loop. A Wait state plus a Choice state checking whether a job has finished bills a state transition on every lap. Thirty-second polling over a four-hour Glue job is 480 transitions for one job run, and it is entirely unnecessary: the .sync integration waits for free. If an option describes a polling loop, it is describing the thing .sync exists to remove.

The sleeping Lambda. A function that waits for an external system holds a billed invocation for the whole wait, is capped at 15 minutes, and cannot survive a deployment. The callback pattern replaces it with a paused state that is not billed for waiting and can wait as long as the workflow's timeout allows.

Both of these are correct in the sense that they work, which is what makes them good distractors. The reason they are wrong is cost and operability, and a good answer says which.

Key terms: AWS Step Functions, Express workflow, Distributed Map, Task token (callback pattern)