GenAI observability in CloudWatch
Lesson 2 of 4 in Monitoring Models and Agents in Production.
Classic monitors assume tabular features and a label that eventually arrives. A foundation model emits text — there is no confusion matrix for "the chatbot got weirder in March". So FM monitoring splits into two layers: operational signals you can meter continuously, and quality signals you sample and judge.
The operational layer is familiar CloudWatch material with new units: latency — end-to-end and time-to-first-token for streaming UX; token throughput — input/output token counts and tokens per second, because tokens are both the capacity unit and (next module) the cost unit; errors and throttles — a throttled FM looks "down" to users while every instance metric stays green.
| Signal | Metric to chart | What a bad week looks like |
|---|---|---|
Latency | End-to-end latency and time-to-first-token (p99, not average) | TTFT creeps up; streaming feels dead before the first word lands |
Token throughput | Input/output token counts, tokens per second, invocations | Prompts quietly bloat; output tokens double after a prompt-template change |
Errors | Invocation errors and throttling counts | Throttles spike at peak hours — capacity, not code, is the bug |
Quality | Sampled evaluation scores, guardrail intervention counts | Judged relevance drifts down after an upstream retrieval change |
For the first layer, the exam's named tool is CloudWatch generative AI observability: pre-built views for GenAI workloads with end-to-end prompt tracing across components — model, knowledge base, guardrail, tool — plus curated metrics and logs. It ships two dashboards: Model Invocations (usage, token consumption, invocation logs with input/output content) and Bedrock AgentCore agents (the next lesson). It works with Bedrock natively and with frameworks like Strands Agents, LangChain, and LangGraph via the ADOT SDK.
The quality layer is Amazon Bedrock evaluations run in production, on a schedule: sample real prompts, run an evaluation job — automatic metrics, an LLM-as-a-judge scoring quality dimensions with explanations, RAG evaluations for retrieve-and-generate, or human review for the stakes that warrant it — and trend the scores. A falling judged-quality trend is the FM equivalent of accuracy drift, caught without waiting for users to complain.
Key terms: CloudWatch generative AI observability, Amazon Bedrock evaluations, Amazon Bedrock
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.