Offline to online: why green suites still ship regressions

Lesson 5 of 5 in Regression Suites in CI: Evals That Run On You.

Your suite is green. Thresholds held, must-pass cases passed, the judge tier improved two points. You deploy, and the complaints start.

This is not a failure of the discipline — it is the boundary of the instrument. An offline suite measures your agent against the cases you thought to write down, and production is the set of cases you did not. The gap has five recurring causes, and each has a specific mitigation. Work through them before you conclude the suite lied to you.

1 · Distribution shift — your dataset is last quarter’s traffic

The cases came from real production runs, which makes them real — and historical. This is distribution shift, the same phenomenon that degrades any deployed model: the inputs move away from the sample you measured on. Product launches, seasonal patterns, a new integration partner, a pricing change: the mix moves, and the suite keeps grading yesterday.

Mitigation: continuously sample new production runs into the dataset, and weight cases by how often that situation actually occurs. A 50-case suite where 40 cases represent 2% of traffic is measuring a corner.

2 · Real users are weirder than your fixtures

Production brings typos, three intents in one message, a 40-page attachment, a language you did not plan for, half a conversation pasted from another vendor’s chat window, and people deliberately probing the system. Hand-written cases are implicitly cooperative — the author knew what the agent was supposed to do.

Mitigation: harvest the ugliest real inputs on purpose. Every escalation, every abandoned session, every “that is not what I asked” becomes a case. Adversarial inputs go into the must-pass set.

3 · Environment drift — the tools return different data now

Offline, your tools hit a fixture database with 200 tidy rows. In production they hit a system with millions of rows, real duplicates, rate limits, timeouts, partially migrated records and a table that changed shape last Thursday. The agent’s reasoning was fine; its inputs were not.

Mitigation: include realistic-scale fixtures and error-injection cases (timeout, empty result, duplicate match, oversized result) — and remember that a green suite says nothing about a schema change in a system you do not own.

4 · You optimised for the suite (Goodhart’s law arrives on schedule)

Twenty iterations of prompt editing against the same 50 cases produces a prompt that is excellent at those 50 cases. Some of that generalises; some of it is a fitted curve.

Mitigation: keep a blind holdout — cases nobody tunes against, looked at only to check whether the tuned gains transferred. When the tuned set improves and the holdout does not, you have measured your own overfitting, which is a genuinely useful result.

5 · You optimised for the judge

If a judge grades the subjective tier, the prompt can drift toward what the judge rewards — more structure, more hedging, more confident phrasing — while human satisfaction stays flat or falls.

Mitigation: periodically re-anchor the judge against human labels, keep the judge prompt versioned, and pair judge scores with at least one signal the judge cannot see: escalation rate, task abandonment, repeat contacts.

The bridge across the gap is shadow evaluation: run the candidate build against real production inputs, in parallel with the live agent, with its side effects disabled. Real distribution, real weirdness, real data — zero user impact. Then grade the shadow runs with the same tiers you use offline, and diff them against what the incumbent did on the same inputs.

Shadow evaluation is where an offline suite’s claims get tested against reality before anyone is affected by them. It is also where the discipline of the previous lessons pays off: the graders, thresholds and report diffs already exist, so pointing them at mirrored traffic is a configuration change rather than a new project.

A shadow run with live side effects is not a shadow run

The candidate must execute with writes disabled or mocked and outbound actions stubbed — no emails, no refunds, no tickets, no webhook calls. Duplicated side effects from a “harmless” parallel run are one of the more embarrassing ways to cause a production incident while trying to prevent one.

Two more constraints worth planning for. Mirroring real traffic means real customer data enters another execution path, so the shadow environment inherits every access-control and retention obligation the primary has. And a shadow run cannot measure what the user would have done next: it tells you what the candidate would have chosen, not whether the customer would have been satisfied. That answer needs a canary on live traffic, with a kill switch — the subject of agentops · rollout and kill-switches.

Five instruments, in order of how much reality they touch
Instrument	What it can tell you	What it cannot	Cost and risk
Offline golden suite	Did this change regress the behaviours we chose to encode? Reproducible, comparable, gate-able.	Anything outside the dataset — new intents, new data shapes, real traffic mix.	Tokens and minutes. Zero user risk.
Blind holdout	Did the tuned gains generalise, or did we fit the suite?	Still your own historical distribution — just an unpolluted slice of it.	Same as the suite, plus the discipline not to peek.
Shadow eval on mirrored traffic	How the candidate behaves on the real input distribution, including inputs nobody imagined.	User reactions and downstream outcomes — nothing it does reaches anyone.	Double inference cost; needs side effects disabled and the same data controls as production.
Canary on a traffic slice	Real outcomes on a small, real population: completion, escalation, satisfaction, cost.	Rare cases — a 1% slice needs volume and patience before its numbers mean anything.	Real user exposure. Requires a kill-switch and per-slice monitoring.
Online eval on sampled production traffic	Continuous quality after launch: drift, degradation, the day the provider changed something.	Causality — it tells you the number moved, not which change moved it.	Ongoing grading spend; sample rate is the budget dial.

AWS Machine Learning Blog — Automated agent evaluation with Amazon Bedrock AgentCore and GitHub Actions (checked 2026-09-13)

The managed platforms have converged on the same shape this module describes. Amazon Bedrock AgentCore Evaluations scores agent interactions from OpenTelemetry traces collected through AgentCore Observability, uses LLM-as-a-judge by default, and offers three modes: on-demand (your CI job), online (a sampled slice of production traffic), and batch. It works for agents hosted on or off AgentCore Runtime, and AWS documents wiring the on-demand mode into a GitHub Actions workflow — a vendor implementation of the per-commit gate.

Read the three modes against the pyramid: on-demand is your pipeline tier, online is the sampled production tier from the matrix above, batch is your nightly.

Microsoft Foundry docs — Agent evaluators (checked 2026-09-13)

Microsoft Foundry ships built-in agent evaluators the docs describe as working “like unit tests for agentic systems”, returning binary pass/fail, split into system evaluation (Task Completion, Task Adherence, Intent Resolution, Task Navigation Efficiency, Customer Satisfaction) and process evaluation (Tool Call Accuracy, Tool Selection, Tool Input Accuracy, Tool Output Utilization, Tool Call Success). Several were still marked (preview) when this was checked.

That split is the outcome/trajectory distinction under different names — evidence that the two axes are not one vendor’s taxonomy but the shape of the problem. This academy is an educational resource, not vendor documentation: confirm evaluator names, availability and preview status in the current docs before you build a gate on them.

Interactive flashcard deck.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.