A/B tests and anomaly detection

Lesson 4 of 4 in Monitoring Models and Agents in Production.

Monitoring caught drift, the pipeline retrained — is the new model actually better in production? Offline metrics do not settle that; traffic does. On SageMaker AI, the mechanism is production variants: one endpoint hosts both models, and variant weights split live traffic (say 90/10). Every variant reports its own CloudWatch metrics — latency, errors, invocations — and your application logs per-variant outcomes. You can also pin a request to a specific variant with the TargetVariant parameter for targeted tests, shift weights gradually as confidence grows, and roll back by setting a weight to zero — no redeploy. (When you want production input without affecting users at all, that is a shadow test: the challenger receives mirrored traffic but its responses are discarded.)

The same humility applies to operational metrics: static thresholds rot. CloudWatch anomaly detection learns a metric's historical pattern — daily and weekly seasonality included — and draws an expected band; the alarm fires on deviation from learned normal, not on a number you guessed. Conceptually underneath this family sits Random Cut Forest (RCF), the unsupervised anomaly-scoring algorithm (also a SageMaker built-in) that scores how much each point disturbs the learned structure — no labels required. Point it at workflow metrics (invocations, queue depth, step durations, token counts) and it catches the weird Tuesday your thresholds were never written for. Data-distribution change detection is the same idea aimed at inputs — which is where this module began.

Interactive sorting exercise: Which monitoring mechanism answers each situation?

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.