The Eval-First Migration

How we swap models without fear: freeze the golden set before shopping, shortlist by requirements, review every flip, and canary with the rollback criteria already written.

Every team we work with eventually faces the same conversation. The system launched on a flagship model because that was the safe choice under a deadline. A year later the bill has a line item somebody finally read, a cheaper tier from another provider looks credible, and someone asks the question nobody wants to own: could we switch? The fear is rational. A model swap changes every output your system produces, all at once — and most teams have no instrument that would tell them whether the new outputs are fine, degraded, or subtly broken in ways that will surface as support tickets in three weeks.

The pattern below is how we take the fear out. Nothing in it is clever. Its entire force comes from one inversion: the evaluation is built before anyone looks at a candidate model. Freeze the measuring stick first, then go shopping — never the other way around. Everything else — the shortlist, the flip review, the canary, the pre-written rollback criteria — follows from that ordering.

Step 1 — freeze the golden set before shopping. If the system has been running an eval harness all along, this step is free: the Golden set already exists, versioned, with its held-out slice intact. If not, build it now, from real traffic, exactly as Building an Eval Harness lays out — sample from production, label with a Rubric, freeze a version. Then run the harness against the incumbent model several times and record two numbers: the baseline score, and the spread across repeat runs. That spread is the noise band; no candidate delta smaller than it means anything.

The timing is the whole point. A golden set assembled after the candidate is chosen inherits the chooser’s hopes. A golden set frozen before is the one artifact in the whole migration that neither vendor, nor budget pressure, nor anyone’s enthusiasm can argue with. Date it, version it, and treat any edit to it during the migration as a red flag in review.

Step 2 — shortlist by requirements, not headlines. Resist the urge to evaluate everything. We follow the Model Landscape domain’s shortlist discipline: write the requirements one-pager first — must-have capabilities, Context window floor, latency and cost targets, deployment constraints, data-residency lines — then filter the field against it and carry two or three candidates into real evaluation. A long shortlist means the requirements are not doing their job yet.

Two habits keep this step honest. First, read each candidate’s Model card for configuration details, not its benchmark table for reassurance — In Production: Reading Model Evals Critically covers why vendor tables cannot settle your question. Second, price the candidates on your actual traffic shape before anyone falls in love: the Token Cost Estimator with your real prompt-to-output ratios, and the Cloud Model Selection Matrix if the move crosses providers. A candidate that only wins on the price sheet at someone else’s traffic shape is not a candidate.

Step 3 — side-by-side runs, then read the flips. Run every candidate through the same Eval harness on the identical frozen items, alongside a fresh incumbent run. One fairness rule matters more than any other: each model gets its best prompt, within the same adaptation budget. Prompts tuned over months for the incumbent quietly encode its quirks — its preferred Chat template rhythms, its tolerance for your System prompt’s length; running them verbatim on a candidate measures prompt-model mismatch, not model quality. So we timebox the same number of hours of prompt adaptation per candidate, log what was changed, and compare the resulting systems.

Then comes the part teams skip and regret: the per-item flip review. Aggregates can match while behavior churns underneath — Production Evals and Regression Testing calls this out as the core reason gates require human eyes. List every item that flipped pass→fail and fail→pass between incumbent and candidate, and have someone who knows the product read each one. If the Grader apex is an LLM-as-judge, spot-check its verdicts on the flipped items too — Judge bias is never more dangerous than when it is scoring two different models’ phrasing styles against each other.

Step 4 — canary with quality monitoring, not just health checks. The harness said yes; production gets the final vote, because no golden set covers the full distribution of what users actually send. We route a small slice of traffic to the candidate and compare it against the incumbent control on quality signals, not just error rates and latency: Refusal rate, format and parse failures downstream, output-length Drift, user feedback where it exists, and a sampled LLM-as-judge score over paired canary/control responses. Where the platform makes it cheap, a shadow stage first — candidate answers generated on mirrored traffic but never shown — calibrates those monitors before any user is exposed.

Two operational notes that save pain. Pin everything pinnable: every managed platform ships models as named versions with published retirement dates, so the migration should land on a pinned candidate version with any auto-upgrade policy explicitly off — otherwise the model you canaried is not the model you will be running in six months. And re-measure cost during the ramp, not before it: Tokenizer differences change token counts for identical traffic, verbosity differences change output length, and prompt-cache hit rates reset when the prompt template changes — the price-sheet delta and the invoice delta are routinely different numbers.

Step 5 — the rollback criteria are written before the canary starts. Not “we’ll watch it closely.” Written thresholds, a named owner, a tested revert path. The next block is the shape we use.

MODEL MIGRATION CHECKLIST — {{SYSTEM_NAME}}
Incumbent: {{CURRENT_MODEL_AND_VERSION}}  →  Candidate: {{CANDIDATE_MODEL_AND_VERSION}}
Owner: {{MIGRATION_OWNER}}   Rollback lever: {{ONCALL_OWNER}}   Date opened: {{DATE}}

BEFORE SHOPPING (all boxes ticked before any candidate output is viewed)
[ ] Golden set frozen: version {{GOLDEN_SET_VERSION}}, {{ITEM_COUNT}} items, held-out slice untouched
[ ] Incumbent baseline recorded: score {{BASELINE_SCORE}} over {{REPEAT_RUNS}} runs, noise band ±{{NOISE_BAND}}
[ ] Requirements one-pager written: must-haves {{MUST_HAVE_LIST}}, cost target {{COST_TARGET}},
    latency target {{LATENCY_TARGET}}, constraints {{RESIDENCY_OR_DEPLOYMENT_CONSTRAINTS}}

SHORTLIST
[ ] 2–3 candidates chosen against the one-pager, reasons logged: {{CANDIDATE_LIST}}
[ ] Cost modeled on OUR traffic shape (prompt:output ratio, cache hit rate), not the price sheet
[ ] Candidate version PINNED; auto-upgrade policy confirmed OFF

SIDE-BY-SIDE
[ ] Same frozen items for every candidate + fresh incumbent run
[ ] Prompt adaptation timeboxed to {{ADAPTATION_BUDGET}} per candidate, changes logged
[ ] Aggregate delta vs. noise band recorded: {{DELTA_RESULT}}
[ ] Every flipped item read by a human: {{FLIP_COUNT}} flips, review notes at {{FLIP_REVIEW_LINK}}
[ ] Judge verdicts spot-checked on flipped items (if judge-graded)
[ ] Go / no-go decision recorded with reasons: {{DECISION}}

BEFORE CANARY (written and signed before any user sees candidate output)
[ ] Rollback triggers, with numbers: {{ROLLBACK_TRIGGER_LIST}}
[ ] Revert path TESTED — config change only, no code deploy required
[ ] Monitors live and calibrated (shadow stage if available): refusal rate, format/parse
    failures, output-length drift, user feedback, sampled judge score vs. control

ROLLOUT
[ ] Canary: {{CANARY_PERCENT}} of traffic for at least {{CANARY_DURATION}}
[ ] Ramp steps and dates: {{RAMP_SCHEDULE}}
[ ] Cost re-measured against the invoice at each step, not just the estimate
[ ] Incumbent path kept deployable until {{SUNSET_DATE}}; retirement is its own reviewed decision