How data goes wrong in production

Lesson 1 of 4 in Ensuring Data Quality.

This is the one topic in the domain where naming a service first makes you worse at it. "Use AWS Glue Data Quality" is a true answer to a question nobody asked — because the hard part is not running a check, it is knowing which check would have caught the thing that went wrong, and that is a design question you can only answer from the failure modes.

So, the failure modes. Six of them cover almost everything that reaches production, and they divide sharply into two groups.

The loud three, where something breaks and somebody looks:

A hard schema change — a column removed, a type narrowed, so the job throws.
A corrupt or truncated file — an upload that failed halfway, so the parser throws.
A referential break — a foreign key with no parent, so the load constraint rejects it.

The silent three, where everything succeeds and the number is wrong:

A soft schema change — a column renamed, so the old name becomes all nulls. No error anywhere.
Late or partial arrival — the export is 80% complete, or yesterday's records arrive today. Row counts are plausible, totals are not.
A semantic change — amounts switch from cents to dollars, a status code gains a value, a timezone changes. Every row is valid; every total is wrong by a factor.

The silent three are the ones that matter, because the loud three are self-reporting. A pipeline with no quality tooling at all still catches the loud three. Everything you build in this module exists for the other half.

Interactive sorting exercise: Sort each defect by whether the pipeline will tell you about it. The useful discomfort of this exercise is how many of them land on the right.

What to carry out of this lesson

Six failure modes, in two groups. Loud: a column removed, a corrupt file, a referential break. Silent: a column renamed, a partial or late arrival, a semantic or unit change.

The loud ones are self-reporting. A pipeline with no quality tooling catches them, which is why a ruleset built from imagination tends to cover only those.

Every check you write is for the silent three. The three that cover them are a schema contract, a row-count comparison against the source, and an aggregate or range check that notices a magnitude shift.

A crawler set to update the table on every run is drift automation, not drift protection. It follows the upstream mistake faithfully.

Put the checks at the boundary, so the verdict arrives before anything downstream can read the data.

Key terms: Schema drift, Late-arriving data, Data quality dimension, AWS Glue Data Quality

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.