The tooling survey — and the portability layer under it
Lesson 3 of 5 in Observability in Production: Watching Agents at Scale.
Before the survey, the decision that makes the survey survivable: instrument once, choose backends later. Emit OpenTelemetry spans from your agent, keep a thin mapping layer between your code and the exporter, and a backend change becomes a configuration change plus a dashboard rebuild — instead of re-instrumenting an agent you no longer remember writing.
That is the whole portability argument, developed properly in Trace anatomy: the conventions are still marked Status: Development, they now live in their own repository, and adopting them anyway beats inventing a private vocabulary nobody else can read. What follows is what the receiving end looks like as of September 2026 — an educational snapshot, not vendor documentation, and not an endorsement.
Langfuse
Open-source LLM engineering platform — tracing, LLM-as-a-judge and code evaluators, prompt management, datasets and a playground. The core is MIT-licensed except the ee folders, self-hostable via Docker Compose or Helm, with a paid managed cloud for teams that would rather not run it.
It is the strongest default when self-hosting is a requirement rather than a preference: a permissive core licence, a documented deployment path, and integrations across OpenAI, LangChain, LlamaIndex, LiteLLM, the Vercel AI SDK and Mastra plus its own Python/JS SDKs.
On portability it does the thing you want: it receives OpenTelemetry traces on an OTLP endpoint (/api/public/otel, HTTP JSON or protobuf — no gRPC as of this check), states that it aims to be compliant with the OTel GenAI semantic conventions, and additionally understands OpenLLMetry, OpenLIT, OpenInference and MLflow attribute flavours. Reading four rival flavours is a feature for you and a fair summary of how unconverged this ecosystem still is.
Snapshot: platform release v4.35.0 (2026-09-11). Note the server and the SDKs version independently — the OTel-native SDK line is its own v4.
LangSmith
LangChain’s commercial observability and evals platform — explicitly not open source. Tracing, dashboards and alerts, online evaluations, and annotation queues, offered as cloud, hybrid, or self-hosted deployment.
Two things make it worth evaluating even if you never touch LangChain. First, it is framework-agnostic: the documented integrations include OpenAI, Anthropic, CrewAI, the Vercel AI SDK and Pydantic AI, and it works without LangChain or LangGraph. Second, the product is built around the production loop rather than the notebook — online evaluations and annotation queues are first-class, and the docs describe an Engine capability for detecting and diagnosing recurring production issues.
Where it fits: teams that want alerting, online judging and human annotation in one managed product and are comfortable buying it. If you are building on LangGraph, the integration depth is the obvious draw — but do not let that be the whole argument, because the lock-in you should worry about is in your dashboards and judge prompts, not your traces.
Hedge: self-hosted deployment is documented; which pricing tier gates it was not verified for this snapshot — confirm with current LangSmith pricing before you plan around it.
Braintrust
Commercial, hosted eval and observability platform, positioning itself as “the active observability platform for instrumenting, understanding, and improving agents.” It combines logging and tracing with experiments and evals, playgrounds, datasets and human annotation in a single workflow.
The centre of gravity is different from a pure tracing tool: Braintrust is built for the iterate-and-compare motion — run a change against a dataset, diff the scores, promote or discard. Production logs feed that loop rather than living in a separate viewer, which is exactly the shape you want when the point of observability is improving the agent and not only explaining it. The docs also describe a Loop agent that builds scorers, datasets and dashboards for you, and a bt CLI for scripting the workflow.
Where it fits: product teams whose bottleneck is eval throughput — many prompt and model variants, needing scores and human review side by side — and who are content on a hosted platform.
Hedge: this snapshot did not verify self-hosting options or the licence status of the surrounding open-source scorer libraries. Check the current docs if either matters to you.
Arize Phoenix
Arize’s open-source AI observability platform for experimentation, evaluation and troubleshooting — built on OpenTelemetry, vendor-, language- and framework-agnostic by construction, and self-hostable. Auto-instrumentation ships through the companion OpenInference project, with openinference-instrumentation-* packages for Python, JS, Java and Go.
Read the licence carefully: Phoenix is Elastic License 2.0 (ELv2), not a permissive OSI licence. For most internal deployments that is a non-issue; if your legal review treats ELv2 differently from MIT or Apache-2.0, or you intend to offer it as a service, that distinction decides the evaluation before any feature does.
Where it fits: polyglot shops that want one OTel-native, self-hosted place where the same spans drive the waterfall view, the evals, the datasets and the experiments — with no second instrumentation layer. The multi-language auto-instrumentation is a genuine differentiator when your agents are not all Python.
If you are already standing on a cloud agent platform, some of this is wired for you — with the platform’s assumptions baked in. Both major offerings below are OpenTelemetry-based, which is the reassuring part: the ingestion path is standard even where the product surface is not.
AWS — AgentCore Observability
AgentCore Observability emits telemetry in OpenTelemetry format to Amazon CloudWatch, viewable on the CloudWatch GenAI Observability page. There is one setup trap worth knowing before you debug an empty dashboard: a one-time, per-account enablement of CloudWatch Transaction Search is required before traces and spans appear.
Agents hosted on AgentCore Runtime are auto-instrumented — OTel-compatible traces, runtime metrics (invocations, session count, latency, errors, CPU and memory) and structured logs. Agents hosted elsewhere, including on-premises and on other clouds, can send telemetry to the same dashboards using the AWS Distro for OpenTelemetry (ADOT) SDK with SigV4 auth against the CloudWatch OTLP endpoint; supported third-party instrumentation includes OpenInference, OpenLLMetry, OpenLit and Traceloop.
The documented telemetry model is a three-tier hierarchy — session (a complete user conversation), trace (one request-response cycle within it), span (a discrete operation: LLM call, tool invocation, memory lookup, inter-agent handoff) — using OTel GenAI semantic conventions and W3C Trace Context propagation. At GA (2025-10-13) AWS also listed integrations with Dynatrace, Datadog, Arize Phoenix, LangSmith and Langfuse, so “AgentCore” and “your existing backend” are not exclusive choices.
Hedged: service surfaces, quotas and console layouts on this platform have moved repeatedly. Treat this as a September 2026 snapshot and check the AgentCore Developer Guide before you design around any specific detail. CloudWatch bills observability ingestion, storage and query separately from the agent runtime.
Azure — Microsoft Foundry observability
Foundry frames observability as three capabilities: evaluation (built-in evaluators for quality, RAG metrics such as groundedness, safety, and agent-specific metrics like tool-call accuracy and task completion), monitoring (real-time dashboards via Azure Monitor Application Insights), and tracing (distributed tracing of LLM calls, tool invocations and agent decisions, built on OpenTelemetry).
Mechanically: you connect an Application Insights resource to the Foundry project, and traces land there under the OTel GenAI semantic conventions, viewable either in the portal’s Traces view or in Azure Monitor. Documented framework support spans LangChain, LangGraph, the OpenAI Agents SDK and Microsoft Agent Framework. Content capture is opt-in, as covered in lesson one.
Agents built and hosted outside Foundry can be registered in a Foundry project (preview) for observability and evaluation: exported spans carry a gen_ai.agent.id attribute matching the registration’s otel_agent_id, and trace-based evaluations run directly over that telemetry. Note the boundaries — human evaluation, trace-to-dataset conversion and AI red teaming were not supported for external agents at the time of this check.
Hedged: the GA/preview line runs straight through this feature set — tracing for prompt and hosted agents is GA while tracing for workflow and external agents remains preview. Preview surfaces carry no SLA and change. Verify current status on Microsoft Learn before committing.
| Your binding constraint | What it points at | What to verify before committing |
|---|---|---|
Traces must never leave our infrastructure | Self-hostable platforms: Langfuse (MIT core, | Read the licence, not the README badge — ELv2 is not an OSI permissive licence, and “open-source core” usually means some features live in the paid or |
We are polyglot — Python, TypeScript, Java, Go agents | Phoenix + OpenInference ships auto-instrumentation across all four; OTLP ingestion elsewhere means you are not blocked, but you may be hand-rolling spans. | Check that the auto-instrumentation covers your actual frameworks in each language, not just the language itself. Hand-rolled spans in the language nobody owns is where trace quality quietly dies. |
Our bottleneck is eval throughput, not tracing | Braintrust or LangSmith — both put experiments, datasets and human annotation next to the logs; Langfuse also ships judge and code evaluators plus datasets in its open core. | Whether your scorers are expressible, whether human annotation fits your reviewers’ workflow, and how the judge model is versioned. A judge you cannot pin is a moving ruler. |
We are already committed to one cloud agent platform | AgentCore Observability → CloudWatch or Foundry → Application Insights. Both are OTel-based and both can ingest telemetry from agents they do not host. | Check preview-vs-GA status per feature, per-account setup prerequisites (Transaction Search on AWS), and the observability bill — ingestion, storage and query are metered separately from the agent runtime. |
We expect to change our mind within a year | Any of them — but only if you keep instrumentation OTel-native behind a thin mapping layer, and treat dashboards, judge prompts and alert thresholds as the assets you will have to rebuild. | That the backend genuinely ingests standard OTLP for your signals. Transport and attribute names port; saved queries, judge prompts keyed to specific fields, alert thresholds and retention rules do not. |
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.