Monitoring Quality in the Wild
Lesson 2 of 3 in Production Evals and Regression Testing.
The gate catches regressions on cases you already knew to test. Production is where the cases you didn’t imagine live — so after a change ships, the eval discipline continues with different instruments. The cheapest ones are online signals at the model level: quantities computable from live outputs, on every request, with no grader in the loop.
Four earn a permanent dashboard. Refusal rate — the share of responses declining to answer — should be stable for stable traffic; a climb means the model, prompt, or safety stack now treats the same questions differently. Format-parse failures — when downstream code parses model output (JSON, tool arguments, structured fields), the parser itself is a free, deterministic grader running on all traffic in real time; its failure rate is one of the most honest signals you will ever get. Length drift — output lengths for similar requests creeping up (verbosity, cost, latency) or down (clipped, thinner answers). And user feedback loops — thumbs, edits, retries, regenerations, abandonment. Feedback is sparse, biased toward extremes, and slow to accumulate, so read it as a qualitative directional signal, never as a score.
| Signal | What it catches | Lag |
|---|---|---|
Refusal-rate trend | Over- or under-refusal after a model, prompt, or safety-stack change — the same traffic suddenly treated differently | Near-real-time once enough comparable traffic accumulates |
Format-parse failure rate | Broken JSON or schema outputs; quiet formatting regressions that crash or corrupt downstream code | Immediate — the parser grades every single request as it happens |
Output length drift | Verbosity creep (cost and latency inflation) or shrinking answers (truncation, thinning quality) on similar requests | Hours to days — needs enough traffic to average over |
User feedback (thumbs, edits, retries) | Quality problems no automatic signal captures: wrong, unhelpful, or off-tone answers that still parse and don’t refuse | Days to weeks — sparse, biased toward extremes, qualitative |
Sampled offline re-grading | Slow drift the per-request signals miss — the harness’s graders re-score a sample of real traffic against your rubric | Whatever your schedule says — it is a batch job you own |
The last row deserves its own paragraph. Sampled offline re-grading pulls a random sample of production traffic on a schedule and runs it through the same Grader ladder the harness uses — code checks first, LLM-as-judge with its known biases managed where judgment is required, against the same Rubric. It is the bridge between the frozen harness and living traffic: the harness tells you how the system performs on yesterday’s photograph of your users; re-grading tells you how it performs on this week’s. (If your system is an agent — a model calling tools in a loop — observability over whole trajectories is its own discipline, taught at our sister AI Agent Academy; here we stop at the model boundary.)
Re-grading is also how you split the most confusable pair in monitoring: input Drift versus behavior drift. Input drift means the traffic changed — new languages, new intents, longer documents — while the system behaves exactly as always; quality drops because questions moved outside what you built and tested for. Behavior drift means the same inputs now get different outputs — a model updated, a dependency changed, a prompt template regressed. The fixes are completely different: input drift calls for refreshing the Golden set and possibly re-adapting the system; behavior drift calls for finding what changed and re-running the gate.
Why online metrics and offline evals disagree
Sooner or later the dashboard and the harness contradict each other: offline scores flat, users unhappy — or feedback souring while re-grades look fine. The instinct is to ask which one is lying. Usually neither is. They measure different populations with different instruments, and four mechanisms drive them apart.
Distribution shift — the photograph vs the movie
The golden set is a photograph of traffic on the day you froze it; production is a movie that keeps playing. Offline numbers describe the photograph faithfully forever — and describe this week’s traffic less faithfully every week. The gap between them is input drift, measured.
Selection effects — who generates the signals
Feedback is not a random sample. The users who click thumbs are unusual; complaints cluster on visible failures and spare the boring ones; re-grades triggered by alerts oversample exactly the traffic that looked bad. Online metrics describe the traffic that emits signals, not the traffic you serve. A doubled thumbs-down rate can mean quality fell — or that a UI change altered who bothers to click.
Grader–user mismatch — two definitions of “good”
Your Rubric measures what you wrote down; users reward what they actually value — speed, tone, brevity, feeling understood. Both are “quality”, but they are different projections of it. A change can improve every rubric line and still land worse, because the rubric never mentioned the thing users cared about.
Censoring — the silent exits
The strongest negative signal is a user who leaves and never comes back — and that user emits no thumbs-down, files no complaint, appears in no re-grade sample. The most decisive verdicts on quality are systematically missing from the feedback data.
The operational takeaway: treat online–offline disagreement as information, not as a bug in one of the instruments. It almost always points at a gap in the golden set (shift), a bias in the feedback path (selection, censoring), or a rubric that has fallen out of sync with what users value (mismatch). Each of those has a concrete fix — and all of them start with sampling random production traffic, because randomness is the only cure selection effects respect.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.