Evaluating agents: task completion and tool effectiveness
Lesson 3 of 3 in Evaluating GenAI Systems.
An agent fails differently from a chat model: the transcript can read beautifully while the wrong refund went out through a real tool. Scoring the final message misses everything that matters, so an agent performance framework has to score two layers — outcomes (was the goal state actually reached?) and trajectories (were the steps that reached it correct, efficient, and safe?) — which is exactly the split the AIP-C01 exam guide asks for. The headline outcome metric is Task completion rate, measured on a suite of tasks with checkable end states; the step layer is Tool-usage effectiveness plus reasoning-quality assessment of the multi-step plan.
| Metric | What it measures | How it is computed |
|---|---|---|
Share of tasks whose goal state was verifiably reached (ticket filed, refund correct) | Replay a golden task suite; verify end states programmatically or with a judge | |
Right tool, valid arguments, successful calls, no redundant or destructive extras | Scored per step from execution traces — tool-selection patterns, argument schemas, call results | |
Reasoning quality | Does the plan follow from the goal; do steps follow from results — or did it luck out? | Judge scores the recorded trajectory (plan, revisions, reactions to tool errors) against a rubric |
Efficiency | Steps, tokens, latency, and cost per completed task | Trace metrics aggregated per session — completion that costs 40 tool calls is a failure wearing a medal |
Scoring an agent: from task suite to regression gate
- Golden task suite
Tasks with programmatically checkable end states — "the ticket exists with priority P2", not "the agent sounded helpful". Hard cases and past incidents live here, same as any golden dataset.
- Run the agent
The candidate agent executes the suite on its real runtime with sandboxed or mocked side-effecting tools, so a bad candidate cannot file real tickets.
- Capture traces
Every session recorded: spans per model call and tool call, token usage, tool-selection patterns — AgentCore observability emits this automatically for hosted agents.
- End state correct?
- Count the completion
Completions aggregate into the task-completion rate — the scorecard headline.
- Audit the steps
Completed or not, the trajectory gets scored: tool choice and argument validity per step, judge-rated reasoning, efficiency. Failures get a diagnosis; successes get checked for wasteful or dangerous paths.
- Aggregate the scorecard
Completion rate, tool effectiveness, reasoning, efficiency — tracked per agent version, exactly like model eval scores.
- Gate the agent change
New tool definition, new prompt, new model under the agent — nothing promotes unless the scorecard holds. The quality gate from the previous lesson, with tasks instead of prompts.
The trace layer is not homework you write yourself. AgentCore observability instruments hosted agents automatically (and other runtimes via the OpenTelemetry-compatible ADOT path), feeding CloudWatch generative AI observability: an Agents view, a Sessions view, and a Traces view where each trace shows the full trajectory and timeline — every model call, tool call, duration, and token count as spans. Metrics land in the bedrock-agentcore CloudWatch namespace. Those traces are exactly what the scorecard consumes: tool-selection patterns for Tool-usage effectiveness, per-session token and latency rolls for efficiency, recorded trajectories for the judge. Instrument first; evaluation is a query over what you captured.
Key terms: Task completion rate, Tool-usage effectiveness, Bedrock AgentCore Observability, CloudWatch generative AI observability, LLM-as-a-judge
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.