Traces, evaluations and the version you can roll back to

Lesson 4 of 5 in Microsoft Foundry Agents in Practice: Identity, Boundaries, Guardrails, Traces, Bills.

An agent failure is rarely reproducible on demand, so the only durable record of what happened is the one you captured while it happened. Foundry organises this into three capabilities, and they answer three different questions:

Evaluation — is it good? Built-in evaluators cover quality, RAG metrics such as groundedness, safety, and agent-specific metrics including tool-call accuracy and task completion. Monitoring — is it healthy right now? Real-time dashboards through Azure Monitor Application Insights. Tracing — what exactly did it do on run 4,812? Distributed traces of model calls, tool invocations and agent decisions, built on OpenTelemetry standards.

Notice that the agent-specific evaluators are the ones you could not have bought for a chatbot. "Did it call the right tool" and "did it finish the task" are questions about control flow, which is precisely the thing the model decides and you did not write.

Three observability capabilities, three questions (Microsoft Learn — Observability in generative AI, checked 2026-09)
Capability	The question it answers	Where it lives	What you must do first
Evaluation	Is this agent good enough to promote — and did the change help?	Foundry evaluators: quality, RAG metrics like groundedness, safety, plus agent metrics such as tool-call accuracy and task completion.	Assemble a dataset of representative tasks. Evaluators score runs; they do not decide what "good" means for your product.
Monitoring	Is it healthy, right now, in aggregate?	Real-time dashboards via Azure Monitor Application Insights.	Connect an Application Insights resource to the Foundry project — nothing lands anywhere until you do.
Tracing	What did this specific run actually do, step by step?	Traces stored in Application Insights using OpenTelemetry semantic conventions for generative AI; viewable in the portal Traces view or in Azure Monitor.	Same connection, plus a deliberate decision about whether to record message content (see the warning below).

Microsoft Learn — Set up tracing in Microsoft Foundry (checked 2026-09)

Foundry stores agent traces in Azure Application Insights using OpenTelemetry semantic conventions for generative AI. You connect an Application Insights resource to the Foundry project and read traces in the portal Traces view or in Azure Monitor.

Two facts that shape architecture. First, because the spans follow the GenAI semantic conventions, your agent telemetry is queryable next to the rest of your application telemetry — the trace is not trapped in a vendor console. Framework support documented as of September 2026 includes LangChain, LangGraph, the OpenAI Agents SDK and Microsoft Agent Framework. Second, capturing chat message content is opt-in (for example AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED, or EnableSensitiveData in .NET) precisely because that content may contain personal data.

Per the GA feature-readiness table checked in September 2026: tracing for prompt and hosted agents is GA, while tracing for workflow and external agents remains preview — confirm current status before you depend on it.

Agents you build and host outside Foundry can still be registered in a Foundry project (preview) to get observability and evaluation. The mechanism is worth knowing because it is pure OpenTelemetry plumbing: spans exported to the connected Application Insights carry a gen_ai.agent.id attribute that matches the registration’s otel_agent_id, and trace-based evaluations then run directly over that telemetry.

Read the limits as carefully as the capability. As of September 2026, human evaluation, trace-to-dataset conversion and AI red teaming are not supported for external agents — and, from the previous lesson, registering an external agent in the Control Plane does not put it behind Foundry guardrails. Registration buys you a pane of glass, not a policy.

Content recording is a privacy decision, not a verbosity setting

Turning on message-content recording is what makes a trace debuggable — and it is also what turns your Application Insights workspace into a store of user conversations, tool arguments and retrieved document text. That data inherits the workspace’s retention, access model and export paths, not your agent’s.

Decide it deliberately, per environment. Full content in development and staging, sampled or redacted in production, retention set to what an incident investigation actually needs, and access to the workspace treated as access to the underlying data. If your Standard setup pulled conversation state into your own Cosmos DB for custody reasons, remember that content-recorded traces are a second copy — and it lives somewhere else.

Now the operational half of this lesson: versions. Foundry snapshots agent versions automatically as you iterate, with rollback and comparison supported, and a published agent’s stable endpoint serves an administrator-selected active version. The REST API additionally supports version_selector rules — for example FixedRatio with a traffic_percentage — so you can pin traffic to one version or split it across two.

That is the whole progressive-delivery toolkit, and it matters more for agents than for ordinary services. A prompt edit is a behaviour change with no compile step and no diff a reviewer can fully reason about; the only trustworthy verdict is measured. Which makes the pairing obvious: evaluations decide whether a version is better, version_selector decides how much traffic finds out, and traces explain the runs that went wrong.

Promoting an agent version without betting the whole endpoint on it

Change instructions, tools or model
Foundry snapshots a new version automatically as you iterate — the artefact exists whether or not you planned a release.
Run evaluators on a task dataset
Quality, groundedness, safety, tool-call accuracy, task completion. Compare against the current active version, not against your memory of it.
Better on the metrics that matter?
Define "matter" before you look. A version that gains 3 points of task completion and loses 10 of groundedness is not an improvement.
Route a slice of traffic (version_selector, FixedRatio)
A traffic_percentage split lets real users exercise paths your dataset never contained — while the majority stays on the known-good version.
Watch traces and monitoring on the slice
Look at loop length, tool-call patterns and failed runs, not just latency and error rate. Agent regressions hide in behaviour, not in HTTP status codes.
Healthy on real traffic?
Make it the active version
The published endpoint stays stable; the administrator-selected active version changes behind it.
Roll back to the previous version
Rollback is the cheap operation here — which is the argument for splitting traffic in the first place.

Tool: Trace Debugger — Read a real agent trace span by span — where the loop went wrong, and which span proves it — in the Trace Debugger.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.