Shadow mode for agents

Read-only replay

How: log production inputs (the initial goal plus any user turns), then re-run the candidate against them offline. Effectful tools are stubbed to return a plausible success; read tools hit real data with a read-only credential.

Good for: the first pass, immediately after CI. Cheap, safe, batch-schedulable overnight, and it works on last week’s traffic so you get volume on day one.

Weak at: anything where the result of an effect changes the next step. If create_ticket returns a stubbed id and the agent later reads that ticket back and finds nothing, the rest of the trajectory is fiction — and the diff you review is fiction too.

Sandboxed side effects

How: the candidate runs against a parallel world — a cloned database or a per-run branch, a mail sink that accepts and never delivers, a payments sandbox key, a filesystem in a container (sandboxing is the same machinery you built for untrusted tool output).

Good for: multi-step effectful workflows, which is most of the interesting ones. Effects are real enough to be read back, so trajectories stay coherent to the end.

Weak at: fidelity drift and cost. Sandboxes go stale, sandbox APIs behave differently from production ones under error conditions, and maintaining the clone is real ongoing work. Budget for it or the sandbox quietly becomes a lie you trust.

Live mirror, effects gated

How: mirror live traffic in real time. The candidate runs concurrently with the incumbent; read tools are live, effectful tools return a recorded "would have" entry instead of executing.

Good for: latency and concurrency realism, and catching interactions with the state the incumbent is currently mutating. Closest thing to the truth.

Weak at: the two versions racing over shared state — the candidate can read a row the incumbent just changed and diverge for reasons that have nothing to do with the version. Also doubles inference spend in real time, so scope it to a sampled percentage of traffic, not all of it.

Tool-call sequence — the shape of the work

Reduce each run to its ordered list of tool names: search → read_doc → read_doc → answer. Now you can count shapes across a few hundred runs and compare the two versions' distributions.

What it catches: a candidate that stopped calling search_kb before answering policy questions (silently ungrounded answers that still read beautifully). A candidate that now calls read_doc nine times where the incumbent called it twice — the same answer, three times the cost. Loops that repeat a pair of calls until the turn limit.

Step count and turn budget — the loop-length distribution

Plot steps-per-completed-task for both versions. You are looking at the tail, not the mean: a candidate whose median is 6 steps and whose 95th percentile jumped from 14 to 40 has learned to flail on hard inputs.

What it catches: the most common quiet regression from a prompt edit. Median unchanged, tail exploded, and your cost and latency live in the tail.

Terminal state — how runs end

Bucket every run: completed, escalated to a human, refused, errored, hit the turn limit. Compare the mix.

What it catches: a candidate that swapped 4% of escalations for 4% of confident completions. That is either the best change you shipped this quarter or the worst, and the terminal-state mix is the only place it shows up before customers find it. Always read the trajectories behind a shift in this mix by hand.

Side effects proposed — the "would have" ledger

Because effectful tools are stubbed or sandboxed, you have a clean record of every action the candidate wanted to take, with arguments. Diff those against the incumbent’s actual actions.

What it catches: the scariest class. A candidate that would have refunded €4,000 where the incumbent refunded €40. A candidate that would have emailed a wider recipient set. Argument-level diffs on effectful calls deserve human review every time, however good the aggregate scores look.

Cost and latency per task — the two numbers finance asks about

Tokens in, tokens out, cached vs uncached, wall-clock per task, and model calls per task, both versions, same inputs.

What it catches: a candidate that is 3% better on quality and 60% more expensive — a real trade-off that a quality-only diff hides. Note the caveat from the ladder: shadow cost measured on replayed traffic understates real cost, because real sessions run longer and messier than logged ones.

Final-answer quality — the judge, used last

Where the pairing is genuinely comparable, run a pairwise LLM-as-judge on the two final answers with position-swapping, and treat the score as one signal among six.

What it catches: substance regressions the structural metrics miss. But it is the noisiest instrument here, and it cannot see cost, effects, or loops — which is why it comes last in this list, not first.