Gates, quarantine, and backfill
Lesson 4 of 4 in Ensuring Data Quality.
Two questions remain, and both are about placement rather than about rules.
Where does the gate belong? Check after publication and consumers can read the bad rows while you deliberate. Check before writing and you cannot run the checks that need the written form — the row count of what actually landed, the partition it landed in, a reconciliation against the target. The resolution is a pattern with a name: write-audit-publish. Write to a staging location, audit what was actually written, and publish with an atomic switch. Apache Iceberg makes the switch cheap — write to a branch, audit the branch, fast-forward main to it — and the same shape works with a staging prefix plus a partition swap, or a staging table plus a view repoint. Two properties worth naming when a scenario asks for them: consumers see either the old data or the new data and never a half-written state, and a failed audit is a no-op rather than a cleanup.
What happens when bad data has already landed? Every real pipeline eventually needs the answer, and it has three parts: quarantine so nothing is discarded, backfill so the window can be rebuilt, and notification so the people who already acted on the number find out. The cost nobody plans for is the third: every hour a known-bad partition stays visible is another hour of extracts, reports and decisions built on it — and fixing the table does not fix those.
Quarantine is the third outcome, and a pipeline with only two is a pipeline that will eventually discard something.
Rows failing row-level validation go to their own prefix or table, tagged with which rule they broke and when, while the rest continue downstream. A 0.3% defect rate stops blocking a daily load, and the 0.3% is still there to fix and replay. Three things make a quarantine real rather than a bin nobody opens:
- The failure reason is stored with the row. A quarantined row with no reason is a row somebody will have to re-diagnose from scratch in three weeks.
- There is a metric and an alarm on its size. An empty quarantine and a broken quarantine look identical otherwise, and the broken one is the dangerous one.
- There is a defined path back. Reprocess into the main table idempotently once the upstream fix lands — which means the path back is a backfill, and it needs to exist before you need it.
And one decision that is easy to skip and is itself a quality matter: do downstream consumers know the batch was partial? A published total that silently excludes quarantined rows is a wrong number, not a safe one. Either publish the exclusion alongside the total, or treat a quarantine above some threshold as a blocking failure.
What makes a backfill possible — and what makes one impossible
A backfill is only ever as easy as the pipeline allowed it to be, and four properties decide that.
The write must be idempotent at partition granularity. INSERT OVERWRITE on one date, or a MERGE on a business key — never INSERT INTO. Appending into an already-populated partition is how a correction becomes a duplication.
The coordinate must be a parameter. A job that reads "today" from the system clock cannot be backfilled at all, because there is no way to ask it for last Tuesday. This is the single commonest reason a backfill turns into a code change under incident pressure.
Parallelism must be bounded. A rebuild of 900 partitions at unbounded concurrency starves the live pipeline and meets a ceiling. Distributed Map with a maximum concurrency is the standard shape.
The rebuilt range must be audited before it is republished — the same write-audit-publish discipline, because a backfill is a write like any other, performed under time pressure by a tired person.
The two kinds of backfill, and why they have different risks
A corrective backfill replaces data that was wrong. A historical backfill computes new logic over old data.
They share the mechanics and differ in one way that matters: a corrective backfill has a known-bad window that consumers may already have read, so notification is part of the job. A historical backfill has no such window — nothing was wrong — so it is pure compute, and its only real risk is starving the live pipeline.
That difference changes the order of operations. For a corrective backfill, quarantine first so the defect stops spreading, then notify, then rebuild. For a historical backfill, write into a staging location, audit it, and publish when it is complete — there is no urgency at all, which means there is no excuse for skipping the audit.
Communicating a restatement, which is the half that is usually skipped
Yesterday's number changed. Somebody will notice, and if they notice before you tell them, you have a second incident about trust.
Three things a restatement notice needs: the window (which dates are affected), the direction and magnitude (totals will rise by roughly 4%), and what to do about artifacts already produced (the Tuesday board pack used the old figure; here is the corrected one).
This is also where lineage earns its keep: downstream lineage turns "we think probably nobody" into a list of datasets, dashboards and subscribers. And it is the argument for a visible as-of or version marker on published datasets — a consumer who can see which version they read can answer "was I affected" without asking you.
When restating is the wrong answer
Not every correction should change history, and a scenario sometimes rules it out.
If downstream consumers have already reported externally — a regulatory filing, a published financial figure, an invoice — then silently restating the source is worse than leaving it, because the two will never reconcile again. The pattern there is a correction record: keep the original period immutable, book the adjustment into the current period with a reason code, and let a consumer choose which view they want.
The same applies to late-arriving data past a stated lateness bound. If you promised that a day is closed at 02:00 the following morning, late records go into the current period as corrections rather than reopening a closed one — and that promise, written down, is what makes the decision a policy rather than an argument.
Key terms: Write-audit-publish, Quarantine, Backfill, Idempotent pipeline step, Late-arriving data
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.