Glue Data Quality and Deequ

Lesson 3 of 4 in Ensuring Data Quality.

Now the tooling, and it is worth knowing its ancestry because the ancestry explains the shape.

Deequ is an open-source library Amazon developed and runs internally: it computes data-quality metrics over a Spark DataFrame, verifies declared constraints against those metrics, profiles datasets, and suggests constraints from what it finds. It is implemented on Apache Spark and designed for datasets of billions of rows.

AWS Glue Data Quality is built on top of Deequ — AWS states this directly — which is why the vocabulary lines up so exactly. Deequ's analyzers are Glue's analyzers. Deequ's constraints are Glue's rules. Deequ's constraint suggestion is Glue's recommended ruleset. Understanding that gives you two useful things: the shape of the managed product makes sense (metrics first, assertions over metrics second), and you know where the escape hatch is — when a check cannot be expressed in DQDL, Deequ or PyDeequ inside a plain Glue or EMR Spark job is the same engine without the wrapper.

Glue Data Quality then offers two places to run, and they answer different questions:

  • Against a Data Catalog table, on a schedule or on demand — a standing health check. "Is this table in good shape?" Results go to the Data Quality results store and optionally to CloudWatch.
  • Inside a Glue ETL job, through the EvaluateDataQuality transform — a gate. "May this batch proceed?" This is the one that can fail a run or route failing rows aside, and it is the one that stops bad data rather than reporting on it.

The second is the one most scenarios are asking for, because reporting on bad data that already published is a different job from preventing it.

A blocking ruleset

Five rules, and every one of them means do not publish. Short on purpose — this is the ruleset attached to stopJobOnFailureOptions.

Rules = [
    ColumnExists "order_id",
    ColumnDataType "amount" = "decimal",
    IsComplete "order_id",
    Uniqueness "order_id" > 0.999,
    RowCount > 50000
]

Note the shape of each one. The first two are the schema contract — vacuous-looking, and they catch the commonest silent defect. IsComplete on the key is absolute. Uniqueness is 0.999 rather than 1.0, because the feed has at-least-once delivery and a nightly false alarm trains everybody to ignore the real one. RowCount is the crude partial-arrival guard; the better version is dynamic.

An observational ruleset

Everything you want a trend of, publishing to CloudWatch and a dashboard, reviewed weekly rather than at 03:00.

Rules = [
    Completeness "email" > 0.95,
    ColumnValues "currency" in ["GBP", "USD", "EUR"],
    ColumnValues "qty" between 1 and 500,
    ColumnLength "postcode" between 5 and 8,
    DistinctValuesCount "channel" <= 8,
    (DataFreshness "order_date" <= 36 hours) OR (ColumnValues "order_date" = NULL)
]

Analyzers = [
    RowCount,
    Completeness "email",
    AllStatistics "amount"
]

Two things to notice. The freshness rule is compound, because DataFreshness fails for NULL rows and pre-orders legitimately have none — the documented remedy. And the Analyzers block gathers statistics with no verdict at all, which is what later supports observations and what tells you where to set the next threshold.

Dynamic and cross-dataset rules

The two forms that stop a ruleset going stale.

Rules = [
    # compare against this metric's own recent history
    RowCount > avg(last(5)) * 0.8,
    Sum "amount" between avg(last(10)) * 0.7 and avg(last(10)) * 1.4,

    # compare against another dataset
    ReferentialIntegrity "customer_id" "customer.{id}" = 1.0,
    RowCountMatch "order_lines" >= 0.99,
    AggregateMatch "sum(amount)" "sum(src_amount)" >= 0.999,

    # anything the built-in types cannot express
    CustomSql "select count(*) from primary where status not in (select code from ref_status)" = 0
]

A dynamic rule compares against the historical values of the same metric, so it tracks a growing business instead of needing a quarterly edit. The cross-dataset rules are the accuracy tier from the previous lesson, and AggregateMatch is the reconciliation that catches a partial arrival and a unit change — the two defects nothing cheaper can see.

The transform, in a job

The call itself, with the four outputs and both option blocks.

from awsgluedq.transforms import EvaluateDataQuality
from awsglue.transforms import SelectFromCollection

RULES = """Rules = [ IsComplete "order_id", ColumnValues "amount" between 0 and 10000 ]"""

evaluated = EvaluateDataQuality().process_rows(
    frame=orders_dyf,
    ruleset=RULES,
    publishing_options={
        "dataQualityEvaluationContext": "orders_boundary_gate",
        "enableDataQualityCloudWatchMetrics": True,
        "enableDataQualityResultsPublishing": True,
    },
)

outcomes = SelectFromCollection.apply(dfc=evaluated, key="ruleOutcomes")
rows     = SelectFromCollection.apply(dfc=evaluated, key="rowLevelOutcomes")

Then split rows on its outcome column: passing rows to the curated table, failing rows to the quarantine prefix with the reason retained. Add stopJobOnFailureOptions only on the blocking ruleset. And set dataQualityEvaluationContext to something meaningful, because it is what names the check in the CloudWatch metrics.

Three neighbours get confused with this tooling in exam scenarios, and the distinctions are clean.

AWS Glue DataBrew is visual, no-code data preparation with one-click profiling — missing-value rates, cardinality, distributions, correlations, outliers. Reach for it when the requirement is an analyst, exploring, without writing code, or when you want a profile of an unfamiliar dataset before you decide what rules to write. It is not a gate in a pipeline.

A Glue crawler infers schema and partitions from data. It is not a quality control, and configured to update the table on every run it will faithfully follow an upstream mistake — which makes it the opposite of a schema contract.

Amazon Macie finds sensitive data — personal data, credentials — which is a different question from whether data is correct. A scenario about discovering unexpected personal data in the lake is Macie; a scenario about a column being null is Glue Data Quality.

And the honest note on effort: a ruleset costs a Spark pass, and the cross-dataset rules cost a join. That is why the accuracy tier usually runs nightly over a closed window rather than on every micro-batch, and why the blocking ruleset is deliberately small.

Key terms: AWS Glue Data Quality, Deequ, EvaluateDataQuality, DQDL, Observation, AWS Glue DataBrew

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.