Model versioning: pinning, upgrades, and silent drift
Lesson 3 of 5 in Deploying and Versioning Agents: Ship It Like Software.
Of the five tuple elements, the model is the one you do not control. The provider ships new snapshots, retires old ones, and — if you let it — decides when your agent changes.
You have three postures, and only one of them is a decision rather than a habit. Pin a dated snapshot and upgrade on your schedule. Float on a family alias and inherit whatever the provider points it at. Or route across a set of models you have each evaluated, which is pinning several models at once plus a policy for choosing between them.
Pinning is the default for anything with a golden suite behind it, for one reason: a pinned model makes "the agent changed" and "we changed the agent" the same sentence. Floating trades that away for whatever the provider improved — and for a bill and a latency profile that can move overnight.
| Posture | What you get | What it costs you | Use it when |
|---|---|---|---|
Pin a dated snapshot | Reproducible behaviour, an honest version tuple, upgrades on your calendar, a known-good target to roll back to. | You own the upgrade work, and you carry deprecation risk: snapshots are retired on the provider’s timetable, not yours. | Anything regulated, anything with an eval gate, anything where an unexplained Tuesday is an incident. This is the default. |
Float on a family alias | Free quality and cost improvements; no upgrade project; never surprised by a retirement. | Uncontrolled change: prompt sensitivity, tool-call formatting, verbosity, latency and spend can all move with no commit and no eval run. | Prototypes, internal tools, and low-stakes assistants where a behaviour shift is an annoyance rather than an incident. |
Route across an evaluated set | Per-task cost/quality matching, and a ready-made fallback path when one provider degrades. | You now evaluate and version N models, and the router itself becomes part of the tuple — its policy is a behaviour change waiting to happen. | High volume with mixed task difficulty, or a hard availability requirement that one provider cannot meet alone. |
Now the part most teams get wrong: a model upgrade is a migration, not a config edit. It has the same shape as a database migration — a new artifact, a compatibility question, a test suite, a staged cutover, and a rollback plan you write before you start.
The migration test is your regression suite from the evals domain, run at n per case because a single run tells you nothing about a distribution. But pass rate alone is too coarse: two models can score identically and behave completely differently. Diff the trajectories as well — which tools each version called, in what order, how many turns, how many tokens, how often it escalated. A silent trajectory change is a future incident that has not chosen its date yet.
The model-upgrade gauntlet
- Provider ships a new snapshot
Or announces a retirement date for the one you are on — same gauntlet, different urgency.
- Pin it as a candidate version
A new manifest, a new version number. The incumbent stays deployed and untouched.
- Run the golden suite, n runs per case
Same cases, same parameters, same tools. Only the model id differs — that is what makes the comparison mean anything.
- Pass rate within tolerance?
Compare against the incumbent’s measured rate, not against an aspiration. Watch the variance, not just the mean.
- Diff trajectories, cost, latency
Tool sequences, turn counts, token spend, escalation rate, refusal rate, output shape. Equal scores with different paths still means different behaviour.
- Differences explainable and acceptable?
- Canary or shadow on live traffic
A small traffic share, or shadow mode where the candidate answers in the dark and a judge compares. Watch the online metrics you cannot simulate offline.
- Online metrics hold?
Human override rate, escalation rate, task completion, cost per resolved task, p95 latency.
- Ramp traffic to the new version
And keep the incumbent deployable until the ramp has survived a full business cycle — including the Monday morning peak.
- Stay on the incumbent; record what broke
A failed upgrade attempt is a valuable eval artifact. Add the failing case to the suite so the next candidate is tested against it.
- New version is the active version
What drift actually looks like in your dashboards
Two signatures. A step change — Tuesday looks nothing like Monday — usually means an input moved: an alias repointed, a tool description changed, a corpus was re-indexed. A slow slide — pass rate down two points a month — usually means your traffic moved: new customer segments, new phrasing, longer documents, seasonal cases the suite never had.
Same symptom, opposite fixes. The step change is a version problem; the slide is a dataset problem, and the answer is to refresh the suite with the traffic you actually get now.
Signals cheap enough to watch on every run
You do not need a judge on every request. These are nearly free and move early:
- mean and p95 output length, and turn count per run
- tool-call validity rate — malformed arguments and schema violations are the first thing to move when a model changes
- refusal and escalation rate
- tokens and cost per completed task (not per call — per task)
- p50/p95 latency by model id
Every one of them needs the version id on the trace to be interpretable. Aggregate metrics without a version dimension cannot answer the only question you will ask.
The daily canary eval
Run a small slice of the golden suite — ten to twenty cases — against the production configuration on a schedule, and alert on movement beyond the noise band you measured. It is the cheapest early-warning system in agent operations, and it catches provider-side and dependency-side change that offline CI (which runs only when you commit) structurally cannot.
Managed platforms are growing this natively: as of September 2026, AWS documents AgentCore Evaluations running in two modes — on-demand scoring of a session against expected responses, expected tool trajectories, and behavioural assertions for CI-style regression testing, and online evaluation that continuously samples live traffic at a configurable rate. Check current docs for the mechanics; the pattern generalises to any stack.
Is it their change or yours?
Answer this with data, not memory. Because the version id is stamped on every trace, you can slice the metric by version and see whether the break aligns with your deploy or with nothing at all. If your last deploy was three weeks ago and the metric broke on Tuesday, stop reading your own diff.
Practitioners widely report behaviour shifts on model endpoints they believed were fixed — serving stacks, safety filters, and defaults around a snapshot can all change without the snapshot id changing. Treat that as a phenomenon to monitor rather than a documented vendor promise, and let your canary tell you rather than an argument on a status page.
What to do the moment you detect drift
Do not start editing the prompt. Prompt-patching a drifted model in production is how one incident becomes two versions nobody can explain.
Sequence: freeze version changes; reproduce with the suite at n runs; quantify the delta; then choose deliberately — pin to an alternate snapshot you have already evaluated, fail over down the chain below, or accept the new behaviour and re-baseline the suite. Whatever you choose, it ships as a new version, through the same gate as everything else.
The fallback chain — degrade in a planned order
- Primary — the pinned snapshot — the version your evals describe
The model in the manifest, at the parameters in the manifest. Every number you quote about your agent — pass rate, cost per task, p95 — is a statement about this tier and nothing else.
- Same-family alternate — previous known-good, already evaluated
The snapshot you were on last quarter, or the smaller sibling in the same family. Cheapest failover because prompts and tool-call formatting usually transfer — but it is only a fallback if you have run the suite on it and know its numbers.
- Cross-provider equivalent — a different vendor, evaluated in advance
Survives a provider-wide outage or a regional capacity problem. Expect prompt and tool-calling differences: a cross-provider fallback that has never run your suite is a second incident stacked on the first.
- Degraded deterministic path — no model, reduced service
Serve retrieval results without generation, return the templated answer, or enqueue the task for later. Often the best user experience available — most people prefer a slower correct answer to a fast confident wrong one.
- Fail closed, and say so — refuse, escalate, stop
For write-capable agents this is a feature, not a last resort. If no tier can be trusted, do not act: hand the task to a human queue and surface the degradation honestly. Wire this to the same kill switch you use for incidents.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.