Evaluation pipelines and quality gates
Lesson 2 of 3 in Evaluating GenAI Systems.
A model evaluated once, at launch, is a model unobserved forever after — every prompt tweak, model-version bump, and Knowledge Base re-sync afterward changes behavior silently. The engineering answer is continuous, not one-off: a golden dataset as the fixed yardstick, regression testing that re-scores every change against it, and an automated quality gate in the release pipeline that blocks promotion when scores fall. Treat it exactly like a test suite: unit tests gate code merges; evaluation jobs gate prompt and model rollouts. Watch one change ride through the gate.
| Strategy | The question it answers | Mechanics | Watch out |
|---|---|---|---|
Offline regression | Did this change degrade anything we already got right? | Evaluation job on the fixed golden set, thresholds gate the pipeline | Only as good as the set — recycle production failures into it |
Canary release | Is the new version safe on real traffic? | Small live slice (say 5%) on the candidate, automated rollback on alarms — error rate, latency, hallucination-rate samples | Needs real-time signals wired before the canary starts |
Which version is better for users and the business? | Sustained randomized split, same metrics per arm: feedback, judge scores on samples, task success, cost | Statistical patience — ending it at the first good day is self-deception | |
How would the candidate behave on production traffic — without users seeing it? | Mirrored requests, responses discarded but scored offline | Doubles inference cost while it runs; no user-feedback signal | |
Synthetic probes | Is the deployed system still itself, right now? | Scheduled synthetic user workflows (CloudWatch Synthetics-style canaries) replay golden journeys; responses checked for consistency and Semantic drift | Probes age — refresh them as the product changes |
The AIP-C01 exam guide extends the gate past the deploy button: deployment validation. After every FM update, synthetic user workflows re-run the golden journeys against the live system; sampled responses pass hallucination-rate checks (a judge or grounding check over a traffic sample) and Semantic drift comparison against baselined answers to fixed probes — catching the model that is fluently, confidently different from last week. Alarms from these checks are what canary rollbacks hang on.
The other continuous input is people: thumbs and ratings in the product, feedback forms, and annotation workflows that turn raw complaints into structured judgments. The loop below is what makes feedback compound instead of evaporate.
1 · Capture — make feedback one tap
Thumbs, star ratings, "report this answer" — attached to the exact response id, prompt version, and model id that produced it. Feedback that cannot be joined back to its invocation is an anecdote, not a signal.
2 · Triage — route the signal
Aggregate ratings watch for trend breaks (a prompt release that drops thumbs-up by five points); individual thumbs-down responses queue for review. Noise is real: single ratings are weak, trends are strong.
3 · Annotate — humans turn complaints into labels
Reviewers grade queued responses on your rubric. Amazon A2I manages human-review loops over model outputs; SageMaker Ground Truth manages labeling workforces at dataset scale. Measure reviewer agreement before trusting the labels.
4 · Recycle — close the loop
Annotated failures become new golden-dataset entries (the gate learns the failure forever) and calibration pairs for the judge (does the judge agree with people? drift here silently corrupts every automated score).
5 · Report — tell stakeholders in their language
Dashboards (CloudWatch, Amazon Managed Grafana) and scheduled model-comparison reports: quality trends, A/B outcomes, cost per resolved task. Stakeholders act on "deflection up 9%, hallucination reports down 40%" — not on a BERTScore.
Key terms: Golden dataset, Continuous evaluation, A/B testing (FMs), Annotation workflow, Semantic drift, Canary deployment
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.