Wiring observability: what a trace looks like on AWS

Lesson 4 of 5 in AWS Agents in Practice: Deploy, Authorize, Guard, Observe, Pay.

Agents are non-deterministic, so the failing run you need to explain may never happen again. That single fact promotes traces from nice-to-have to the primary debugging surface: if you did not record it, you cannot reproduce it, and you will be reduced to reading the final answer and guessing.

AWS’s answer is boringly standard, which is the best thing about it. AgentCore Observability emits telemetry in OpenTelemetry format to Amazon CloudWatch, viewable on the CloudWatch GenAI Observability page. Agents on Runtime are auto-instrumented — OTel-compatible traces, runtime metrics (invocations, session count, latency, errors, CPU and memory) and structured logs. Agents hosted anywhere else — on-premises, another cloud — send the same telemetry to the same dashboards using the AWS Distro for OpenTelemetry (ADOT) SDK with SigV4 auth to the CloudWatch OTLP endpoint.

The data model is three tiers, and knowing the vocabulary is what lets you talk to the console instead of at it. A session is a complete user conversation. A trace is one request-response cycle inside that session. A span is a discrete operation inside a trace — an LLM call, a tool invocation, a memory lookup, an inter-agent handoff. AWS uses OTel GenAI semantic conventions and W3C Trace Context propagation, which is why a span from your own service can appear in the same tree as a span from the platform.

So what does a real trace look like? Below is an illustrative span tree for one turn of a support agent — invented for teaching, with plausible timings — followed by the wiring checklist and the symptoms of each wiring mistake.

An illustrative span tree

Illustrative example, not captured output — shapes and names are what matter.

session  sess-8f21  (actorId: user-4417)          total 12.4s
└─ trace  turn-3  "cancel my order and refund it"
   ├─ span  gen_ai.chat  model=claude-*  1.9s   in 4,812 tok / out 118 tok
   ├─ span  memory.retrieve  namespace=/summary/user-4417/  0.3s  -> 2 records
   ├─ span  tool.gateway  lookup_order  0.4s   permit
   ├─ span  gen_ai.chat  2.2s   in 5,904 tok / out 96 tok
   ├─ span  tool.gateway  issue_refund  0.1s   FORBID  (policy: amount > 1000)
   ├─ span  gen_ai.chat  1.7s   in 6,240 tok / out 143 tok
   └─ span  tool.gateway  escalate_to_human  0.5s  permit

Read it as a story: the model looked the order up, tried a refund, was denied deterministically at the Gateway, and escalated. Note the input-token count climbing every turn — that is the cost curve of lesson five, visible in a trace.

The wiring checklist

Enable CloudWatch Transaction Search once per account. No traces or spans without it.
Confirm the emit path. On Runtime, auto-instrumentation covers traces, metrics and logs. Elsewhere, install the ADOT SDK and point it at the CloudWatch OTLP endpoint with SigV4 auth.
Propagate context across your own services. W3C Trace Context is the mechanism; if your API layer drops the header, the tree breaks into orphans.
Decide what a session id means in your product and set it deliberately — Runtime will generate a runtimeSessionId if you do not, and an auto-generated id is one you cannot correlate to a support ticket.
Point evaluation at the same data. Online evaluation samples live traffic from a CloudWatch log group at a configurable 0.01–100% sampling rate, writing results to /aws/bedrock-agentcore/evaluations/results/{config_id} and metrics to the Bedrock-AgentCore/Evaluations namespace.
Then set alarms on things that matter — error rate, session count, latency, and a quality metric such as Builtin.GoalSuccessRate.

Symptoms and causes

Metrics but no traces or spans → Transaction Search not enabled in that account/region. The classic.

Traces for the platform but not your own service calls → trace context not propagated through your API layer, so your spans start new trees.

Nothing at all from a non-Runtime agent → no ADOT instrumentation, or SigV4 auth failing to the OTLP endpoint. Auto-instrumentation is a Runtime convenience, not a universal default.

Online evaluation configured but scoring nothing → it can only use evaluators that need no ground truth. Evaluators referencing expected responses or assertions are on-demand only.

Beautiful traces nobody reads → the real failure. Traces are a debugging surface and the input to evaluation and to escalation design; if no one has a reason to open them, you are paying CloudWatch to store regret.

Four telemetry paths into the same dashboards
Path	What you install	What you get	Watch out for
Agent on AgentCore Runtime	Nothing — auto-instrumented.	OTel-compatible traces, runtime metrics (invocations, session count, latency, errors, CPU/memory) and structured logs.	Still needs the account-level Transaction Search switch for traces and spans.
Agent on Lambda, EKS, on-premises or another cloud	The ADOT SDK, pointed at the CloudWatch OTLP endpoint with SigV4 auth.	The same AgentCore Observability dashboards for an agent AWS is not hosting.	You own the instrumentation quality — spans exist only where you emit them.
Framework-level instrumentation	A supported third-party library: OpenInference, OpenLLMetry, OpenLit or Traceloop.	Framework-aware spans without hand-writing them for every tool call.	Two instrumentation layers can double-count spans. Pick one per component.
Third-party destinations	Your existing vendor integration — at GA, AWS named Dynatrace, Datadog, Arize Phoenix, LangSmith and Langfuse.	Agent telemetry in the tool your on-call already lives in.	Two homes for traces means two retention policies and two bills. Decide which one is authoritative for incidents.

Tool: Trace Debugger — Practise reading agent traces the way you just read that span tree — find the step where the run went wrong, not the step where it visibly failed.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.