Quality gates on the way into the model
Lesson 1 of 3 in Data Pipelines for FM Consumption.
In classic ML, bad data trains a bad model and you find out at evaluation. In GenAI, bad data flows straight into production answers: a corrupt document gets chunked, embedded, retrieved, and quoted back to a customer with full confidence. AIP-C01 therefore puts a validation gate on every path into the model — the RAG corpus, fine-tuning sets, and runtime inputs alike.
The named tooling: AWS Glue Data Quality evaluates datasets against declarative DQDL rulesets (IsComplete, Uniqueness, ColumnValues, ColumnLength…), can recommend rules from a profile of your data, and its anomaly detection learns metric trends over runs — catching the nightly refresh that suddenly delivers a tenth of the usual rows. SageMaker Data Wrangler covers visual exploration and transformation. Custom Lambda functions carry the GenAI-specific checks no generic rule engine ships — encoding sanity, language detection, extraction integrity, token-length budgets. And CloudWatch metrics make the gate observable: pass rates, quarantine counts, alarms. (Deep fundamentals of data quality live on the MLA path; here they aim at FM consumption.)
The gate between raw data and the model
- New batch lands
Documents for the corpus, examples for tuning, or records for inference — every path into an FM deserves the same gate.
- Glue DQ evaluates
A DQDL ruleset (completeness, uniqueness, value domains, lengths) plus anomaly detection on learned metric trends — rules you wrote and drifts you did not foresee.
- Lambda checks GenAI specifics
Encoding and language checks, token-length budgets, extraction integrity, PII pre-screens — the checks DQDL does not express.
- All checks pass?
- Into chunking & embedding
Only validated data reaches the pipeline that feeds the model.
- Quarantine + alarm
Failed batches divert to a quarantine location; CloudWatch metrics record the failure and an alarm notifies the owning team.
- Fix upstream, re-run
The gate's job is to make bad data the producer's problem before it becomes the model's.
| Tool | Shape of check | Reach for it when |
|---|---|---|
Glue Data Quality | Declarative DQDL rules + recommendations + anomaly detection on metric trends | Batch/tabular gates on Data Catalog tables and ETL pipelines |
SageMaker Data Wrangler | Visual profiling and 300+ transforms, exportable as processing jobs | A human needs to see and shape the data before codifying the pipeline |
Custom Lambda validators | Arbitrary code per record or batch | GenAI-specific checks: encodings, token budgets, extraction integrity |
CloudWatch metrics | Pass rates, quarantine counts, alarms on thresholds | Making the gate observable — a silent gate rots |
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.