Metrics, logs, and alarms that matter
Lesson 1 of 5 in Maintaining and Monitoring.
Here is the sentence this lesson exists for: an alarm on job failure cannot detect a pipeline that stopped running.
Follow the mechanism rather than the intuition. A Glue job publishes a failed-task metric when it runs and fails. If the job never starts — the schedule was deleted, the trigger was deactivated, the upstream file never arrived, the account hit a concurrency ceiling and the submission was refused — then no run happens, no data point is published, and the evaluation window is empty. An empty window is not a breach. With CloudWatch's default treat-missing-data setting of missing, the alarm transitions to INSUFFICIENT_DATA and stays there, silent, indefinitely.
So the pipeline is dead, the dashboard is stale, and the alarm is not in ALARM state. It is in the third state nobody puts on a runbook.
There are two correct answers and they are both about measuring something that is always published:
- A heartbeat. Emit a metric on every successful completion and alarm on its absence with
treatMissingData: breaching. Now absence is the signal. - A freshness measurement. A scheduled check computes the age of the newest data and publishes it as a metric, independent of whether the pipeline ran. Now the alarm measures the thing the business actually cares about.
The second is strictly better, because it also catches the case where everything ran successfully and produced nothing.
| Setting | A missing data point counts as | Use it for | The trap |
|---|---|---|---|
| bad | a heartbeat, or anything that should report continuously | it will fire during a legitimate pause, so a maintenance window needs suppression |
| good | a metric that only exists when something goes wrong, such as a throttling counter | if the publisher itself dies, the alarm reports health for ever |
| nothing — the window is scored on the remaining points, and an empty window goes to | intermittent metrics where absence genuinely carries no information | the default, and the reason a failure alarm on a job that never ran is silent |
| nothing — the alarm keeps its current state | a noisy metric where you want state to persist across gaps | an alarm stuck in OK stays in OK through a total outage |
Two more CloudWatch mechanisms belong in a data engineer's toolkit, and both are about turning a counter into something worth alarming on.
Metric math builds the metric no service publishes. Services emit counters; operations cares about ratios and derived quantities. Divide failed runs by total runs to alarm on a failure rate rather than a count, so a pipeline that runs 4,000 times a day does not page on one transient failure. Use FILL to make a sparse metric continuous, which is often a cleaner fix than fighting the missing-data setting. Use SEARCH to aggregate across every job in an account without naming them, so a new pipeline is covered on the day it ships rather than the day somebody remembers to add it to the dashboard.
Composite alarms express a condition over other alarms. Two uses. The first is honest noise suppression: one upstream outage trips thirty downstream alarms, and a composite is what pages while the thirty remain as diagnostic detail. The second is that "the pipeline is unhealthy" genuinely is a conjunction — the job failed and the data is stale and the retry queue is filling — and a composite states that in one place rather than in three runbooks. Composite alarms can also suppress child alarms while a known parent condition is active, which is the documented way to stop a maintenance window generating pages.
What a composite cannot do is see something none of its children can see. If every alarm you own watches the pipeline's activity, a composite of them is still blind to incident three.
Key terms: Treat missing data, Freshness check, Composite alarm, CloudWatch anomaly detection, Metric math
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.