Observing Glue, EMR, and streams

Lesson 2 of 5 in Maintaining and Monitoring.

The previous lesson was about detection. This one is about diagnosis, and diagnosis needs a different kind of instrument: not "is something wrong" but "which stage, which task, which key".

For a Spark job — in Glue or on EMR — there are four surfaces and they answer four different questions. Knowing which one to open is most of the skill:

Continuous logging answers what is it doing right now. Driver and executor output streamed to CloudWatch while the job runs, with a filtered stream that strips framework noise so your own log lines are readable.
Job run insights answers why did it fail. On a failed run Glue writes the failing line of your script, the last Spark action in the plan, a time-ordered sequence of the relevant driver and executor events, and a rule-based root cause with a recommendation.
The Spark UI answers which stage, and which task. Stage timeline, task duration distribution, shuffle read and write volumes, spill to disk.
Observability metrics answer is this normal for this job. Published every 30 seconds across reliability, performance, throughput and resource utilisation.

For a stream, there is really only one question — how far behind are we — and it has a precise metric, which the lesson comes to below. The thing that makes stream monitoring different from job monitoring is that falling behind has a deadline: records expire.

Streams have one question and a deadline.

For Kinesis Data Streams the metric is GetRecords.IteratorAgeMilliseconds — the age of the last record returned, measured against the newest record in the stream, across shards and consumers. Zero means current. The reason it is the first metric to alarm on is the deadline: a stream's retention is finite, 24 hours by default and configurable up to 365 days, and when the iterator age approaches the retention period, records expire unread and are gone permanently. Alarm on the Maximum statistic — the convention is to fire when the age passes half the retention window — because the Average across shards hides a single stuck shard completely.

Two companion metrics complete the picture. WriteProvisionedThroughputExceeded says producers are hitting the per-shard write limit of 1 MB/s or 1,000 records/s, often because the partition key is skewed and one shard is hot. ReadProvisionedThroughputExceeded says consumers are hitting the read side — 2 MB/s and five GetRecords calls per second per shard — which is what enhanced fan-out exists to relieve when several consumers share one stream. The Kinesis Client Library also emits MillisBehindLatest per shard, which is how you find the one shard that is stuck.

On Amazon MSK the same idea wears Kafka clothes: consumer lag in offsets, plus an estimate in time, which is the more actionable of the two because a thousand-record lag means something different at ten records a second than at ten thousand. And note Kafka retention is by time or size, so an unread partition can be truncated by volume long before its time limit.

The shape of the curve is the diagnosis in both cases: a step is a downstream failure or a deployment, a steady climb is a consumer that cannot keep up with current throughput, and a sawtooth that returns to zero between batches is usually fine.

The first metric to alarm on, per component, and the symptom that tells you to look there. The right-hand column is the one worth memorising.
Component	First metric	What it means when it rises	The usual fix
Glue job	run duration, then `glue.driver.workerUtilization`	the window is closing, or you are paying for workers that idle	auto scaling, a different worker type, or addressing skew — not simply more workers
Spark stage	`glue.driver.skewness.job`	one task is far longer than the median, so one executor works alone	adaptive query execution, salt the hot key, broadcast the small side, or filter the sentinel value out
Kinesis stream	`GetRecords.IteratorAgeMilliseconds` (Maximum)	consumers are falling behind, and past half the retention it is a loss countdown	add consumer capacity; raise retention only as a stopgap while you fix it
Amazon MSK	consumer lag, in offsets and estimated time	the same, in Kafka vocabulary	more consumers in the group, or more partitions if the group is already at parity
Lambda consumer	iterator age, plus the dead-letter queue depth	a shard is blocked, usually by a record that always fails	partial batch responses, bisect on error, a record age limit, an on-failure destination

What to carry out of this lesson

Four Spark surfaces, four questions: continuous logging (now), job run insights (why it failed), the Spark UI (which stage and task), observability metrics (is this normal).

Job run insights only exists for a failed run — and the rule-based stream only when a rule matches. A successful run with wrong output is invisible to it.

Skewness reads 0 until the longest task exceeds the stage skewness factor, default 5. Worker utilisation near 0.2 is a sizing problem, not a code problem.

Iterator age is the stream metric, on the Maximum statistic, because an Average hides one stuck shard. Past half the retention window — default 24 hours, up to 365 days — it is a data-loss countdown.

Read the shape: a step is a downstream failure, a steady climb is insufficient consumer capacity, a sawtooth is probably fine.

Key terms: Glue observability metrics, Glue job run insights, Glue continuous logging, Spark UI and history server, Iterator age, Consumer lag

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.