Shadow variants and A/B before you ship
Lesson 4 of 4 in Evaluating Models — Classic and GenAI.
Offline metrics audition a model; production traffic is the real interview. Live requests carry distribution drift, malformed inputs, and latency pressure no curated test set reproduces. SageMaker gives two graduated exposure mechanisms. Shadow testing: the endpoint duplicates a configurable share of requests to a shadow variant whose responses are logged for comparison — and never returned to callers. A/B via production variants: two live variants split real traffic by weights, and real users see both. Shadow answers "does it behave?" at zero user risk; A/B answers "does it move the business metric?" at controlled risk.
| Method | Users see the challenger? | Risk | It decides |
|---|---|---|---|
Offline evaluation | No — curated test data only | None | Is the model plausible at all? (necessary, never sufficient) |
Shadow test | No — responses logged, never returned | None to users; some compute cost | Does it behave on real traffic — quality, latency, errors? |
A/B (production variants) | Yes — real traffic split by variant weights | Controlled: cap the challenger's weight | Does it improve user-facing and business outcomes? |
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.