Observing GenAI: tokens, hallucination rates, agent traces
Lesson 3 of 3 in Performance and Monitoring for GenAI.
Classic observability asks is it up, is it fast, is it erroring. GenAI observability adds a fourth question those three never needed: is it right? The stack answers it in layers. Metrics: the AWS/Bedrock namespace (invocations, token counts, TTFT, throttles) plus your business metrics on custom dashboards. Logs: Model Invocation Logs — the opt-in, full-fidelity record of every request and response body. Traces: end-to-end prompt tracing through CloudWatch generative AI observability and AgentCore Observability, with X-Ray still covering the non-FM hops. And quality signals: hallucination rates, golden datasets, and output diffing — the layer that watches what the other three cannot see.
| The question | The signal | Where it lives |
|---|---|---|
What are we spending, per model, right now? | InputTokenCount / OutputTokenCount by model ID | CloudWatch, AWS/Bedrock namespace (GenAI observability charts them) |
What exactly did the model see and say? | Full request/response bodies with metadata | Model Invocation Logs → S3 (Athena-queryable) and/or CloudWatch Logs |
Which hop added the latency? | End-to-end trace with per-hop spans | GenAI observability prompt tracing; AgentCore traces; X-Ray for non-FM services |
Did the agent misuse a tool — or another agent? | Tool-call spans: arguments, latency, errors; delegation tracking | AgentCore Observability sessions on the CloudWatch AgentCore dashboard |
Is quality drifting? | Judge-scored samples, golden-dataset runs, output diffs | Scheduled evaluation jobs publishing custom CloudWatch metrics and alarms |
Is the vector store healthy? | Query latency/recall, index freshness, ingestion lag | Vector-store metrics + automated index-optimization and data-quality routines |
Playbook: the token bill spiked at 2 a.m.
CloudWatch anomaly detection on token-count metrics catches burst patterns the moment they deviate from the learned band; Cost Anomaly Detection confirms it in billing terms. Then invocation logs answer which prompts: an agent in a retry loop, a scraper, or a runaway batch look completely different in the request bodies.
Playbook: answers got worse but no metric moved
Latency, errors, tokens — all flat; users complain anyway. This is the quality layer's case: compare the week's golden-dataset run against baseline (output diffing), check the judge-scored sample trend, and read reasoning paths on failed sessions to see whether retrieval thinned out, a tool started failing silently, or a model/prompt change shifted behavior.
Playbook: RAG answers degrade as the corpus grows
Watch the retrieval hop as its own service: vector query p99, recall against a labeled set, index freshness versus source updates, ingestion-pipeline lag. Automated index optimization routines (re-sharding, parameter tuning, compaction) and data quality validation on what enters the store keep retrieval from rotting invisibly under a healthy-looking model.
Playbook: compliance asks what the model told a customer
Forensic traceability = invocation logs (the exact bodies, timestamped) + CloudTrail (who called which API) + trace IDs linking the user session to the invocation. If logging was never enabled, the honest answer is "we cannot know" — which is why audit-sensitive workloads enable it on day one and encrypt the destinations.
Key terms: Model invocation logging, Hallucination rate, Golden dataset, Output diffing, CloudWatch generative AI observability, Bedrock AgentCore Observability
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.