Drift: data, concept, and detection pipelines

Lesson 1 of 4 in Monitoring Models and Agents in Production.

A deployed model is a photograph of the world on the day it was trained — and the world keeps moving. Nothing errors, latency stays green, and the predictions quietly rot. The exam calls this drift, and it comes in three flavors you must be able to tell apart from a scenario description:

Data drift — the input distribution shifts. A marketing push brings in a younger customer segment; inflation moves transaction amounts; a mobile release changes how a field is filled in. The relationship the model learned may still hold — it is just being asked about inputs it rarely saw.
Concept drift — the relationship between inputs and the right answer shifts. Fraudsters change tactics, so feature patterns that meant "legitimate" last quarter now mean "fraud". Inputs can look statistically identical while the model's learned mapping goes stale.
Prediction / quality drift — what you observe downstream: the output distribution moves (suddenly approving twice as many loans) or measured accuracy sags once labels arrive. It is the symptom; the first two are causes.

The three drift flavors — and the tell in a scenario question
Flavor	What shifts	Fraud-model example	Needs ground truth?
Data drift	Input feature distributions	A new payment type floods in; amounts and merchant mix look nothing like training data	No — compare live inputs against a training-time baseline
Concept drift	The input → label relationship	Same-looking transactions, but fraud rings changed tactics — old patterns now mean the opposite	Yes — you only see it in label-based quality metrics (or a proxy like attribution drift)
Prediction/quality drift	Output distribution or measured accuracy	Fraud-flag rate halves overnight; precision drops when investigations close	Partly — output distribution no, accuracy yes (labels arrive late)

Detection is a pipeline, not a dashboard glance, and every piece has a SageMaker name. SageMaker Model Monitor defined the pattern the exam tests: enable data capture on the endpoint (request/response payloads copy to S3), run a one-time baselining job over the training dataset to compute statistics and constraints, then schedule monitoring jobs that compare captured traffic against that baseline and emit violations as CloudWatch metrics. Four monitor types map to the drift flavors: data quality (input distributions), model quality (accuracy against ground-truth labels you merge in — which always arrive late), bias drift (via SageMaker Clarify), and feature attribution drift (SHAP ranking shifts — a label-free proxy for concept drift).

The loop closes with automation: violation metric → CloudWatch alarm → EventBridge → retraining pipeline (the MLOps module builds that trigger). Retraining then produces a new baseline — drift is always measured against what the current model learned, not against last week's traffic.

Model Monitor and Clarify moved to maintenance — the concepts did not

AWS has closed SageMaker Model Monitor and SageMaker Clarify to new customers. Both docs carry the same wording, verified October 2026: existing customers continue as normal, AWS keeps investing in security and availability, but no new features are planned.

That pairing is not a coincidence — two of Model Monitor's four monitor types, bias drift and feature attribution drift, are Clarify doing the measuring on a schedule. They went into maintenance together because they are the same machinery.

The MLA-C02 guide still names and tests both, so learn them: baseline from training data, scheduled comparison, the four monitor types, alarm → retrain; and on the Clarify side, pre-training bias metrics, post-training bias metrics, and Shapley-value feature attributions. What "maintenance" changes is the answer to "what would you build today" — on a newer stack the same pattern is assembled from data capture plus scheduled evaluation plus CloudWatch. An exam question asks what the service does, not whether you could still sign up for it.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.