When Models Change Under You

Lesson 3 of 3 in Production Evals and Regression Testing.

You control your prompts, your pipelines, and your golden set. You do not control the vendor’s roadmap. Hosted models get updated, gain successors, and get retired — on the vendor’s schedule, announced in the vendor’s changelog. For a system whose behavior you have carefully gated, this is a standing threat with two defenses.

Pin versions wherever the platform allows. Most managed platforms distinguish a fixed, versioned model from an alias that auto-updates to the newest version. The alias is convenient and quietly deadly: it means a model change can reach production without any action from you — no gate, no baseline diff, no announcement in your own release history. Pinning turns vendor updates back into events you schedule.

Watch deprecation calendars like expiring certificates. Pinned versions do not live forever; platforms publish retirement dates, and when one arrives your traffic moves whether you are ready or not. A deprecation notice is a project deadline with an eval milestone in front of it — and every forced migration is a model swap, which means it goes through the same regression gate as a voluntary one. The only difference is who set the date.

A vendor model-update notice lands — what do you do?

Interactive decision tree — outcomes:

  • Evaluate on your own schedule

    A new version with no retirement date attached is an opportunity, not an obligation. Queue it as a normal candidate: run the harness when the roadmap has room, and let the numbers argue for or against switching.

  • You are evaluating after the fact — compensate deliberately

    With no pinning available, updates land silently. Tighten the online signals from lesson two, re-run the harness whenever the vendor announces any change, and keep behavior baselines fresh — they are your only proof that outputs moved when you suspect they did.

  • Migrate on your schedule, before theirs

    Passing the gate early converts a forced migration into a routine release with slack for surprises. Ship it, keep the old version live as a rollback until its retirement, and store the successor’s outputs as the new baseline.

  • Blocked — remediate before the deadline, and say so loudly

    Adapt prompts and settings to the successor, consider alternative models, and escalate the timeline risk now. The retirement date will not wait for your backlog, and the worst version of this story is discovering the regression during a forced cutover.

The decision tree leans on one artifact you have to create before you need it: behavior baselines. Keep more than scores — store the current model’s actual outputs on the Golden set, per item, alongside the grades. Three payoffs. At migration time, you can diff the successor’s outputs item by item and see not just whether cases flipped but how the answers changed. When you suspect a silent update, replaying the golden set against your stored outputs gives you evidence instead of vibes. And when your Rubric improves later, you can re-grade old outputs under the new rubric and keep comparisons honest across time.

One anti-pattern deserves a name: migrating on release notes. The vendor’s announcement will say the successor is better, and it may well be — on the vendor’s benchmarks, averaged over the vendor’s idea of traffic. None of that answers the only question that matters: what happens on your cases? Never quote a public Benchmark delta as evidence about your task; the harness run is the evidence. That is the entire posture of this module compressed into one habit.

In production

Every managed platform lives with the same tension: vendors must ship new model versions and retire old ones, while customers need behavior to hold still. The mechanisms are the same everywhere — versioned model identifiers you can pin, published deprecation notices with dates, and your own eval-on-migration. What differs is the defaults.

AWS

Amazon Bedrock addresses each hosted model by identifier and publishes a model lifecycle: versions move from active toward end-of-life on announced dates. Pinning means invoking a specific model version and subscribing to lifecycle announcements; a forced migration is a change of identifier that should never ship without a full harness run and a per-item diff against your stored baselines.

Azure

Azure OpenAI in Azure AI Foundry ships models as named versions with per-deployment upgrade policies — a deployment can auto-upgrade to the newer default version or stay pinned to a specific one until its published retirement date. The default is the trap: if you never chose a policy, find out which one your deployments carry, because “auto-upgrade” means model swaps reach production without passing your gate.

Google Cloud

Vertex AI distinguishes stable versioned models from aliases that track the latest release, and publishes discontinuation dates for versioned models. Pinning to a version freezes behavior until that date; the discipline to wire in is eval-on-migration — when a discontinuation notice lands, the first action is a harness run against the successor with per-item diffs, not a calendar reminder for the week before cutover.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.