Built-in tools, Observability, Evaluations, Registry
Lesson 4 of 6 in Amazon Bedrock AgentCore, Service by Service.
Two tools ship with the platform because almost every serious agent needs them and almost nobody should build them: a code sandbox and a browser.
Code Interpreter vignette. A finance analyst asks an agent to reconcile two spreadsheets. The agent writes pandas, runs it in a sandbox, reads the traceback, fixes the column name, runs again. That loop is enormously effective and it means executing code the model just invented — which is only acceptable somewhere that is not next to your credentials.
Browser vignette. A supplier portal from 2009 with no API. The agent drives a managed cloud browser to log in, download the invoice PDF, and file it — and when it stalls on an unexpected captcha, a human opens Live View, takes over for ten seconds, and hands control back.
Code Interpreter
Isolated sandbox environments with pre-built runtimes for Python, JavaScript and TypeScript, session-based file storage, persistent session state for multi-step workflows, and configurable network modes — fully isolated, or controlled public network access. The Python sandbox pre-installs the usual analysis stack (pandas, numpy, matplotlib, scikit-learn, scipy, seaborn, bokeh, sympy, statsmodels).
Files: inline uploads up to 100 MB, and up to 5 GB when transferred to or from Amazon S3 via terminal commands inside the sandbox. Actions are logged to AWS CloudTrail, so "what did the agent execute" is an audit query rather than a guess.
The network mode is the security decision, and it is the one people click past. Isolated means the sandbox cannot phone home — which is precisely the containment you want when the code was written by a model that just read an untrusted PDF. Per-session CPU and memory caps exist; check the Code Interpreter Service Quotas.
Browser
A fully managed cloud browser. Each session runs in a dedicated isolated environment — own CPU, memory, filesystem — terminated and sanitized when the session completes. Agents drive it over WebSocket-based streaming APIs, commonly Playwright over the Chrome DevTools Protocol.
Sessions default to a 15-minute timeout with a maximum of 8 hours, and run concurrently. Observability is built in: Live View real-time streaming where a human can watch and take over, CloudTrail logging, CloudWatch metrics, and — for custom browsers — session recording capturing DOM changes, actions, console logs and network events to S3 for replay in the console.
Two flavours: the AWS managed browser (aws.browser.v1), or a custom browser with session recording, custom network settings and a specific IAM execution role. Browser profiles can persist authentication state across sessions, and corporate proxies are supported with credentials in AWS Secrets Manager.
Observability is where AgentCore stops being AWS-specific in an interesting way. Telemetry is emitted in OpenTelemetry format to Amazon CloudWatch and viewed on the CloudWatch GenAI Observability page. Agents hosted on Runtime are auto-instrumented — OTel-compatible traces, runtime metrics (invocations, session count, latency, errors, CPU and memory) and structured logs, for free.
Agents hosted elsewhere — on-premises, on another cloud, on Lambda or EKS — can send telemetry to the same dashboards using the AWS Distro for OpenTelemetry (ADOT) SDK with SigV4 auth against the CloudWatch OTLP endpoint. Third-party instrumentation libraries including OpenInference, OpenLLMetry, OpenLit and Traceloop are supported, and at GA AWS added integrations with Dynatrace, Datadog, Arize Phoenix, LangSmith and Langfuse. The data model is standard OTel with GenAI semantic conventions and W3C Trace Context propagation, which is why this is portable knowledge rather than a vendor lock-in story.
The three-tier telemetry hierarchy
- Session — a complete user conversation
The outermost unit: everything one user did across many requests. Sessions are how you answer "was this customer’s whole experience broken, or just one turn?" — and they line up with the Runtime
runtimeSessionId, so cost, isolation and telemetry share a boundary. - Trace — one request-response cycle within a session
One turn of the agent loop as the user experiences it: input in, answer out, with everything the agent did in between. This is the unit you sample for evaluation and the unit a user complains about.
- Span — discrete operations inside a trace
The spans are where debugging actually happens: an LLM call, a tool invocation, a memory lookup, an inter-agent handoff. Non-determinism means you cannot re-run the failure — so the span record is the evidence. Capture it or lose it.
Evaluations turns that telemetry into judgment, in two modes that answer two different questions. On-demand asks "did this change break anything?" — score a specific session, optionally against ground truth. Online asks "is production drifting?" — continuously sample live traffic. It works with agents regardless of framework, and as of September 2026 evaluates TypeScript agents built with Strands, LangGraph, OpenAI Agents and the Vercel AI SDK alongside the Python ecosystem.
Built-in evaluators include Builtin.GoalSuccessRate, Builtin.Correctness and Builtin.Helpfulness; AWS counted thirteen built-ins at GA covering response quality, safety, task completion and tool usage. You can add custom LLM-as-a-judge evaluators or custom code-based evaluators as Lambda functions. Results land in CloudWatch logs and are emitted as metrics in the Bedrock-AgentCore/Evaluations namespace — which means agent quality can page someone.
| On-demand | Online | |
|---|---|---|
Question it answers | Did this change regress behaviour? | Is live traffic getting worse? |
Input | A specific session, optionally with | Sampled traffic from a CloudWatch log group |
Ground truth | Supported: expected responses, expected tool trajectories, behavioural assertions | Not available — production has no answer key |
Evaluator constraint | Any evaluator, including judges referencing | Only evaluators that need no ground truth; an online config supports up to 10 evaluators |
Sampling | You choose the sessions | Configurable rate from 0.01% to 100% |
Natural home | CI/CD regression gates against a golden dataset | Dashboards and alarms on live quality |
Agent Registry is the least glamorous service and the one that decides whether any of this scales past three teams. It is a managed discovery service: a centralized, governed catalogue where an organisation publishes MCP servers, tools, agents, agent skills and custom resources, with hybrid search (keyword plus semantic), catalogue browsing, approval workflows for publishing, IAM and OAuth (custom JWT) access control, and a native MCP endpoint so agents and IDEs can query the registry directly.
That last detail is the interesting one: the catalogue is itself an MCP server, so an agent can discover what it is allowed to use at runtime instead of shipping with a hardcoded tool list. It can catalogue agents hosted on AgentCore, on other providers, or on-premises; CloudTrail provides the audit trail; GA in August 2026 added AWS Organizations auto-detection of AgentCore Runtimes and Gateways plus customer-managed KMS keys.
Without something in this slot, every organisation converges on the same failure: four teams build four overlapping "customer lookup" tools, none of them knows the others exist, and two of them are wrong.
Tool: Trace Debugger — Sessions, traces and spans are only useful if you can read them. Debug a failing agent run from its trace in the Trace Debugger.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.