AWS Glue ETL and its job model

Lesson 1 of 5 in Transformation Engines.

AWS Glue is serverless Spark with four things bolted on that plain Spark does not have, and knowing which four is most of what a question tests: the Data Catalog as a metastore, job bookmarks for incremental reads, DynamicFrames for schemas that disagree with themselves, and connections that package credentials and network path together.

The capacity model is deliberately simple. A DPU is 4 vCPU and 16 GB, billed per second with a one-minute minimum per run. A worker type fixes vCPU, memory and local disk together, and each worker node runs exactly one Spark executor whose core count equals its vCPU count. So the two levers you have are more workers (parallelism) and bigger workers (memory and disk per task), and they fix different problems — which is the distinction the next three lessons are built on.

One number to carry about billing: the one-minute minimum per run means a thousand tiny jobs cost a thousand DPU-minutes of floor regardless of the work done. A pattern of one Glue job per arriving file is therefore expensive in a way that is invisible in the job's own runtime, and the fix is to batch the files.

And one limit that bites the same pattern from a different direction: MaxConcurrentRuns defaults to 1. A second run started while the first is still going raises ConcurrentRunsExceededException — unless job queuing is enabled, in which case it waits instead. So a job triggered per arriving file does not merely cost a floor per run; past the first concurrent file it fails, which is why an event-driven Glue pipeline needs either a raised concurrency limit, queuing, or a queue in front of it that batches. Note the interaction with the previous paragraph's advice and with bookmarks: concurrent runs of the same job interleave their bookmark state, so raising the limit on a bookmarked job is not free either.

Worth separating two cost ideas that get collapsed into one wrong rule, because a stem will test the difference.

DPU-hours are DPU count x elapsed time. So for work that divides cleanly, more parallelism is not dearer: 10 workers for 10 minutes and 20 workers for 5 minutes are both 100 worker-minutes, and you have bought elapsed time rather than capacity. What is dearer is parallelism you cannot use — when elapsed time does not fall because the stage is bound by a shuffle or by one skewed task, the capacity-seconds rise for no gain. The test before adding workers is therefore whether the work actually divides, and the simulation in the next lesson measures exactly that.

And one architectural fact that generates support tickets. A job attached to a connection runs its Spark environment with network interfaces in that connection's subnet, so it has that subnet's connectivity and nothing more. A private subnet with no route to Amazon S3 produces a job that reaches the database and then hangs reading its own script — and the fix is an S3 gateway VPC endpoint, which is the single most common Glue networking answer. The security group also needs a self-referencing inbound rule, because Spark executors talk to each other.

The Glue worker ladder, verified against the documentation. One Spark executor per node, with cores equal to vCPU — so "more workers" and "bigger workers" buy different things.
Worker	DPU	vCPU / memory	Disk	Reach for it when
G.025X	0.25	2 vCPU / 4 GB	~84 GB	a streaming job whose volume is small but continuous — the point is running cheaply all day rather than fast
G.1X	1	4 vCPU / 16 GB	~94 GB	the default for ordinary transformation work; start here and move only for a measured reason
G.2X	2	8 vCPU / 32 GB	~138 GB	joins, wide aggregations, or a first response to spill and `Container killed by YARN`
G.4X / G.8X	4 / 8	16 / 32 vCPU · 64 / 128 GB	256 / 512 GB	large shuffles that exhaust local disk, or a driver that must hold a very large file index
G.12X / G.16X	12 / 16	48 / 64 vCPU · 192 / 256 GB	768 GB / 1 TB	the most demanding jobs, on Glue 4.0 and 5.0
R series	same DPU count	memory-optimised at the same DPU	as the G equivalent	memory pressure that is not relieved by more partitions — large keyed state, very wide rows
Python shell	0.0625 or 1	a single process, no Spark	—	a small script that calls APIs or moves metadata and has no business starting a cluster

What a bookmark stores, and the two ways it breaks

A job bookmark is per-job state that remembers what the previous run processed. What it stores depends entirely on the source, and that is where every one of its failure modes comes from.

For Amazon S3 sources it compares the objects' last-modified timestamps, not their names. So a file that is re-uploaded, copied, or touched by a lifecycle transition or a re-encryption gets a new modification time and is processed again — which is also why compacting a partition makes a downstream incremental job reprocess everything it touched. Equally, a file back-dated into an already-processed window is never seen.

For JDBC sources it tracks bookmark keys, by default the primary key, and compares values. The keys must be strictly monotonically increasing or decreasing — a revision number whose values repeat across the dataset is not valid. And the decisive limitation: a JDBC bookmark captures only newly inserted rows. An UPDATE advances no key and is invisible, which is why "use job bookmarks" is the wrong answer to any stem mentioning updates or deletes.

Six documented ways a bookmark that is switched on still reprocesses everything, and they share a symptom: the output is correct and the bill is a multiple of what it should be. A missing job.init() or job.commit(), so the state is never initialised or never advanced. A transformation_ctx that is missing or duplicated across frames. Reading with plain Spark DataFrames rather than DynamicFrames, which bypasses the Glue-side feature entirely. Non-sequential JDBC bookmark keys. And concurrent runs of the same job interleaving their state.

Pause is the under-used third mode — process the increment without advancing state, which is how you re-run a window for testing without losing your place. And note which mode you get for free: Disable is the default, so a job nobody configured is reprocessing its whole source every run by design rather than by accident.

Key terms: AWS Glue, DPU (data processing unit), Glue worker type, Glue job bookmark, DynamicFrame, Glue connection, Flex execution class