Online evaluation and the drift you never deployed
Lesson 4 of 5 in Observability in Production: Watching Agents at Scale.
Your golden dataset describes the tasks you thought of. Production sends the ones you did not: the customer who pastes a whole email thread, the tenant whose product catalogue is in two languages, the request that arrives at 40× normal length. Offline evals tell you the agent still handles yesterday’s cases. They cannot tell you it handles today’s traffic.
Online evaluation closes that gap: sample live runs, score them, and treat the scores as a monitored signal. Three sources feed it, and they are not interchangeable.
Judge grading on a sample. Run an LLM judge over a fraction of production traces — was the task completed, was the answer grounded, was the tool choice sensible. This is the only source that produces a quality number on traffic nobody has labelled. User feedback signals. Explicit thumbs are rare and biased; the implicit ones are richer and nearly free — did the user retry, edit the output, abandon mid-run, escalate to a human, or accept and move on. Drift detection. Comparing today’s distribution against a baseline, which is how you catch the failure mode this lesson is named for.
Now the part that has no analogue in ordinary software. You can regress without deploying. The model underneath you is a dependency you do not version, running on infrastructure you cannot see, and behaviour can change while your repository sits untouched.
That is not a hypothetical: providers roll model versions, retire snapshots, adjust serving configurations, and update safety filters. Your prompts were tuned against a specific behaviour, your judge was calibrated against it, and your tool descriptions were written for a model that read them a particular way. When any of that shifts, the symptom shows up in your dashboards as an unexplained Tuesday.
| Drift source | How it shows up in your dashboards | How you detect it deliberately | Containment |
|---|---|---|---|
Model version rollover — the provider moves you to a new snapshot or retires the old one | A step change on a specific date with no deploy behind it: success rate, output length, tool-call rate or refusal rate shifts overnight. | Record the exact model id and version on every span, then chart your metrics by model version. Without that field you are guessing. | Pin explicit model versions rather than floating aliases where the provider allows it; subscribe to deprecation notices; keep a small canary run set you can replay against a new version before you accept it. |
Silent serving changes — same model id, different behaviour | No step change, just a slow slide in judge scores or a rising rate of one specific failure mode. | A frozen canary set of a few dozen fixed inputs replayed on a schedule at temperature zero. Divergence on a fixed input with unchanged code is the cleanest possible drift signal. | Alert on canary divergence, not just on production aggregates. Then reproduce, and escalate to the provider with the trace pair — this is what your recorded prompts are for. |
Upstream tool or API change — a dependency alters its schema, semantics or error strings | Tool error rate up on one tool, or worse: the tool succeeds and returns subtly different content, so quality drops with no error anywhere. | Contract tests on tool responses plus per-tool success and schema-validation metrics. Track empty or degenerate results as a distinct outcome from errors. | Validate tool outputs against a schema at the boundary; treat validation failures as first-class events; version your tool integrations and monitor per-version. |
Retrieval corpus drift — the knowledge base grows, gets reorganised, or gains a contradictory document | Groundedness or citation-quality scores fall while retrieval latency and hit counts look normal. | Track retrieval score distributions and the age mix of retrieved documents; judge groundedness on sampled runs rather than assuming it. | Re-run your RAG eval set after significant corpus changes, gate large ingests behind that check, and keep document ids in traces so a bad answer is traceable to a bad source. |
User population drift — new tenants, new languages, new task mix | Aggregate success rate falls while every existing cohort is unchanged — the mix moved, not the quality. | Always segment by tenant, locale, and task type. An aggregate that only ever moves for mix reasons is a metric that lies to you weekly. | Add the new cohort to your golden dataset, then decide explicitly whether to support it, restrict it, or route it to a human. |
Prompt and config edits by non-engineers — someone improves the system prompt in a console | A behaviour change with no code deploy and no model change, often noticed first as a shifted output format. | Version prompts as artifacts, stamp the prompt version onto every span, and chart metrics by prompt version. | Treat prompt changes as releases: review, version, canary, and the ability to roll back. Prompt-management features exist in these platforms for exactly this reason. |
Myth: “Judge everything — sample rates are just cost-cutting”
A judge call is a model call. Judging 100% of traffic can approach the cost of serving it, and it buys precision you rarely need — the point of online judging is to move a monitored aggregate, not to grade every run.
Choose the rate from the base rate you need to detect. Roughly: to see a failure mode that occurs in 1% of runs with enough samples to trend it, you need hundreds of judged failures, which at 1% means tens of thousands of judged runs. Low-volume, high-stakes agents may genuinely need 100%; a high-volume assistant does not. Sample stratified, not uniformly: judge more heavily where risk lives — new cohorts, high-value tenants, runs that hit retries or guardrails.
Myth: “User thumbs-down is ground truth”
It is a signal with heavy selection bias. Users rate when they are angry or delighted, so your feedback set is bimodal and unrepresentative, and satisfaction is not correctness — a confident wrong answer often scores well.
Use explicit feedback as a pointer to interesting traces, not as a metric. The implicit signals are usually better behaved: retry rate, edit distance between the agent’s output and what the user actually sent, abandonment mid-run, escalation to a human. Those correlate with real dissatisfaction and arrive on every run rather than on the 2% that click.
Myth: “Online evaluation replaces the offline eval set”
They do opposite jobs. Online evaluation discovers — it finds failure modes on traffic nobody imagined. Offline evaluation locks in — once you have found a failure, you add it to the golden dataset so it becomes a regression test that runs before every release.
The loop is the point: online finds it, offline pins it, the fix ships, and both signals confirm. A team with only offline evals never learns about new failures; a team with only online evaluation re-learns the same failure every quarter.
Fact: your judge drifts too
The judge is a model. It sits on the same provider, subject to the same rollovers and serving changes, and a shifted judge moves your quality metric with no change in agent quality — the most confusing incident in this whole domain, because the dashboard is wrong rather than the system.
Defences: pin the judge model version explicitly, keep a small human-labelled calibration set and re-score it whenever the judge changes, and record the judge model id and prompt version on every score you store. If the calibration set moves, the ruler moved.
Fact: no ground truth means fewer evaluators are available online
Production has no answer key, so evaluators that need one — exact-match correctness, expected tool trajectories, assertion checks — are inherently offline or on-demand. AgentCore encodes this restriction explicitly: online configurations can only use evaluators that require no ground truth.
What still works online: reference-free judging (helpfulness, groundedness against the retrieved context, task completion inferred from the trace), process checks over the trajectory (did it call a tool it should not have, did it ignore the tool output it fetched), and structural checks (schema validity, loop depth, guardrail trips). Design your online suite from that menu rather than trying to smuggle ground truth into production.
Tool: Eval Suite Builder — Build an online evaluation config — pick reference-free evaluators, set a sampling rate against a target base rate, and see what your judge bill and detection latency look like.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.