A troubleshooting order

Lesson 5 of 5 in Maintaining and Monitoring.

Somebody says the data is wrong. You have four services, nine jobs, three caches and twenty minutes. The single biggest determinant of how long this takes is whether you work in an order or follow a hunch — because a hunch starts in the middle, and the middle is where every explanation is plausible.

The order has five questions, and each one eliminates rather than explains. That is the point: you are not looking for the cause yet, you are halving the search space.

Did it run? Not "did it succeed" — did a run exist at all. A deactivated trigger, a deleted schedule, a refused submission and an upstream file that never arrived all land here, and all of them are invisible to a failure alarm.
Did it fail? If a run exists and failed, you are nearly done: job run insights names the line and the cause.
Did it read what you think it read? The bookmark state, the partition predicate, the projected range, the input row count. An enormous share of "wrong data" incidents are a correct job reading the wrong input.
Did it write what you think it wrote? Output row count against input, the partition it landed in, whether it appended where it should have overwritten.
Is the consumer reading what it wrote? The four caches. The pipeline can be blameless and the dashboard still wrong.

Notice that steps 1, 3 and 5 are the ones that produce a green pipeline and a wrong number — and that they are the ones people skip.

"The data is wrong." Work it in order.

Interactive decision tree — outcomes:

It is not your pipeline
The input never arrived. Confirm against the landing prefix or the stream, then tell the upstream owner — and afterwards add the freshness or arrival alarm that would have told you this before the consumer did. This is the most common "pipeline incident" that is not a pipeline incident.
The trigger, not the job
A deactivated Glue trigger, a deleted schedule, an EventBridge rule disabled during an unrelated change, a notification configuration overwritten. Check CloudTrail for configuration changes in the window: this class of incident has a who and a when, which makes it the fastest of all to close once you look in the right place.
A ceiling, not a bug
Submissions were refused with a concurrency exception, or they are sitting in a queue. Check for ConcurrentRunsExceededException in the caller’s logs and whether job queuing is enabled. The fix is in the ceilings lesson: fan in, bound the fan-out, or queue deliberately.
Read the insights streams
A failed run is the easy case. Job run insights gives the failing script line, the last Spark action and a rule-based root cause with a recommendation; the Spark UI gives the stage and task if the cause is resource-shaped. Resist the urge to retry before reading, because a retry that succeeds hides the evidence.
It read less than you think — four usual suspects
A bookmark that advanced past unprocessed rows. A partition predicate that excluded the data. A projected range that does not cover the date, returning zero rows silently. A partition that exists in S3 but was never registered in the catalog. All four produce a correct job over a wrong input.
It read more than you think — duplicates upstream
An object rewritten in place and re-read by the bookmark. Two concurrent runs of the same bookmarked job. An at-least-once notification delivered twice. A source that replayed a block of records. Reconcile the input count against the source for a closed window — that reconciliation is a DQDL RowCountMatch rule waiting to be written.
A join fanned out
Output exceeding input with no explicit explosion is almost always a join against a dimension that is not unique on the join key — a slowly changing dimension with overlapping validity windows is the classic. Count distinct keys on both sides. A Uniqueness rule on the dimension key would have caught it at the boundary.
It wrote somewhere else
The rows exist, in the wrong place: a partition value derived from the wrong timestamp, a path built from the system clock instead of the batch parameter, an append where an overwrite was intended. Note that a job which reads "today" from the clock also cannot be backfilled, so this bug and that limitation have one cause.
The pipeline is innocent — it is a cache
The source of truth is right, so work the four caches outward: the Redshift result cache (cannot be stale), a materialized view awaiting refresh, Athena result reuse within its max age, a SPICE refresh that failed silently. Then put an as-of timestamp on the dashboard, because this incident will recur.

Three habits make the difference between working this order in twenty minutes and working it in a day.

Establish the blast radius before the cause. As soon as you know which dataset is wrong and for which window, ask the lineage graph who consumes it. Notification is parallelisable; diagnosis is not, and the people downstream would rather know at minute five with no explanation than at minute ninety with a good one.

Do not retry before you read. A retry that succeeds destroys the evidence and converts a diagnosable failure into a recurring mystery. Capture the run id, the input and output counts, and the insights streams first. This is also why idempotent steps matter for investigation and not only for correctness: if a rerun is safe, you can afford to leave the failed state alone and reproduce it deliberately.

Reconcile a count, every night, automatically. The three incident classes that produce a green pipeline and a wrong number — read less, read more, wrote more — are all count defects, and a nightly comparison of input rows, output rows and the source's own count for a closed window catches all three. That check is the bridge into the final module: it is not monitoring, it is data quality, and it is the cheapest rule anybody ever writes.

The parallel track: what to do while you are still diagnosing

A consumer reports a wrong number
Record the exact claim: which dataset, which column, which window, which figure they expected. Half of all "the data is wrong" reports are resolved at this step because the expectation itself was wrong.
Establish the affected dataset and window
Not the cause — the scope. Which table, and from when to when. This is the input to everything that happens in parallel, and it is usually available in two minutes from the consumer’s own query.
Ask lineage who else consumes it
Downstream lineage turns "we think probably nobody" into a list. Do this before you know the cause, because notification does not depend on diagnosis and the people downstream are making decisions right now.
Tell the consumers, with the window and no theory
"The curated orders table is suspect for 18–21 October; do not rely on it; we will update by 15:00." A theory offered at minute ten is usually wrong and will be quoted back for a week.
Stop the spread: pause downstream, or quarantine the window
Pause the dependent jobs, or mark the affected partitions so downstream reads exclude them. The aim is that the defect stops propagating while you work — every hour it stays visible is another hour of extracts and decisions built on it.
Work the five-step order
Did it run, did it fail, did it read right, did it write right, is the consumer reading it. In that order, eliminating rather than explaining.
Fix, then reprocess the window
Correct the cause, then backfill the affected range idempotently — partition overwrite or merge, never append — and audit the rebuilt range before republishing it.
Convert the incident into a rule
Whatever you checked by hand to confirm the defect becomes an automated check: a count reconciliation, a uniqueness rule on the join key, a freshness alarm. An incident that leaves no rule behind will happen again, and the next person will not have read this page.

Key terms: Glue job run insights, Freshness check, Backfill, Idempotent pipeline step, OpenLineage

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.