X-Ray, AgentCore Observability, dashboards
Lesson 2 of 4 in Cost and Performance Engineering.
When an ML system misbehaves, the exam expects you to reach for the right layer of a three-tool stack. CloudWatch is the floor: metrics, logs, alarms, and dashboards for everything — endpoints publish invocation and latency metrics automatically, containers write to CloudWatch Logs. AWS X-Ray answers the question CloudWatch cannot: which hop in a request chain (API Gateway → Lambda → SageMaker endpoint → DynamoDB) is adding the latency — distributed traces assembled into a service map. And for agent workloads, Amazon Bedrock AgentCore Observability records what the agent did: sessions, step-by-step traces, tool invocations, and token usage per AgentCore primitive (runtime, memory, gateway, built-in tools, identity), surfaced on the CloudWatch GenAI observability dashboard.
| Question in the incident channel | Tool |
|---|---|
"Is it slow or erroring right now?" | CloudWatch metrics + alarms |
"What did the container actually log?" | CloudWatch Logs (Logs Insights to query) |
"Which hop in the chain adds the 900 ms?" | AWS X-Ray traces + service map |
"What did the agent do — which tools, how many tokens?" | AgentCore Observability (GenAI dashboard) |
"Is this instance type oversized for its utilization?" | Compute Optimizer (fleet) / Inference Recommender (endpoints) |
A dashboard earns its place by answering "is the model serving its users?" at a glance. For a SageMaker AI endpoint that means: ModelLatency (time inside your container) next to OverheadLatency (time SageMaker spends around it) — the split tells you whether to optimize the model or the plumbing; Invocations and Invocation4XXErrors / 5XXErrors; and GPU/CPU/memory utilization per instance — a GPU endpoint idling at low utilization is money, a saturated one is tomorrow's latency page. Capacity follows the same metrics: endpoint auto scaling (via Application Auto Scaling) target-tracks a metric such as invocations-per-instance between a min and max, while keeping a provisioned floor protects latency through spikes auto scaling cannot catch in time.
ModelLatency vs OverheadLatency — why both tiles?
ModelLatency measures your container answering; OverheadLatency measures SageMaker request handling around it. High ModelLatency → optimize the model, batch size, or instance. High OverheadLatency → look at payload sizes and request patterns. One number alone sends you optimizing the wrong thing.
What belongs on the GPU tile?
GPU utilization and GPU memory utilization per instance. Low utilization on a GPU family is the most common right-sizing finding — the fix is a smaller accelerator, Inferentia, or batching. Persistent near-saturation means scale out before the latency complaints start.
Which metric should auto scaling track?
The one that moves with real load: invocations-per-instance for request-driven endpoints (or concurrency-style metrics for LLM serving). Scaling on CPU alone misleads for GPU-bound models — the CPU can look bored while the GPU is the bottleneck.
Where do Bedrock workloads fit on the dashboard?
Bedrock publishes invocation counts, token counts, latency, and throttle metrics to CloudWatch — chart throttles explicitly. For agents, the GenAI observability dashboard adds session traces and per-tool outcomes.
Interactive sorting exercise: The incident channel asks — which tool do you open first?
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.