Monitoring Models and Agents in Production

Drift in its three flavors and the pipelines that catch it, CloudWatch generative-AI observability, agent failure modes worth paging on, and A/B tests that settle arguments.

Content current as of 2026-10.

Lessons

1. Drift: data, concept, and detection pipelines
2. GenAI observability in CloudWatch
3. Watching agents: tool failures and truncated streams
4. A/B tests and anomaly detection

Sources

• Amazon SageMaker Model Monitor (developer guide)
• Amazon CloudWatch: generative AI observability
• Amazon Bedrock: evaluate the performance of Bedrock resources