Rollback: the state that will not go backwards
Lesson 4 of 5 in Deploying and Versioning Agents: Ship It Like Software.
Rolling back an agent's code is trivial. Point the endpoint at v6, restart, done — the same as any service.
Rolling back an agent's state is where teams discover that rollback was never really about the code. Because while v7 was live it was writing: memory records, conversation histories, checkpoints, cached summaries, rows in other people's systems. Some of that state has a shape v6 has never seen.
The canonical failure is one sentence long: v7 wrote memory that v6 cannot read. You roll back the container in ninety seconds and spend the next four hours watching v6 crash on records with a field it does not know about, or — much worse — silently ignore that field and make decisions on a partial view.
Sort the state an agent touches into four buckets, because each has a different rollback story:
Behaviour state — the tuple itself. Rolls back cleanly; that is what the manifest is for.
Agent-written durable state — long-term memory, extracted preferences, summaries, embeddings, learned routing hints. Rolls back only if the readers are compatible across versions.
Run state — checkpoints, session filesystems, pending approvals, queue messages in flight. Mid-run rollback is a mixed-version execution: v6 code resuming a checkpoint v7 wrote.
External side effects — the refund issued, the email sent, the ticket closed, the row updated in the CRM. These never roll back. They can only be compensated, and only if you built the compensating action.
| What you changed | Rolls back? | What bites you | Make it safe by |
|---|---|---|---|
Prompt, parameters, tool schemas, framework build | Yes, cleanly | Nothing, if the manifest is atomic and immutable. If any element lives outside it, that element does not come back with the rest. | Keeping every tuple element in the manifest and deploying only by manifest. |
Pinned model snapshot | Usually | The old snapshot may already be retired by the provider, in which case there is nothing to roll back to. | Tracking provider retirement dates as dependency expiry dates, and keeping an evaluated alternate on the fallback chain. |
Memory / long-term record schema | No — data outlives code | v6 meets records with new fields, renamed keys, or new record types. Crashes are the good outcome; silent misreads are the bad one. | Schema-versioning every record and shipping tolerant readers one release before the writer changes (expand → migrate → contract). |
Checkpoint / run-state format | No for runs already in flight | v6 resumes a v7 checkpoint and either fails to deserialise or resumes into a step that no longer exists. Framework checkpointers are not required to be backwards compatible. | Stamping the version in the checkpoint and refusing cross-version resume: drain in-flight runs, or let them finish on v7 while new runs start on v6. |
Anything the agent did to the outside world | Never | The email is sent. The refund cleared. The ticket is closed and the customer read it. | Compensating actions where they exist, gates where they do not, and blast-radius limits so a bad version cannot do 10,000 of them before you notice. |
Trap: mixed-version execution — v6 resuming v7’s work
A rollback mid-flight leaves runs whose checkpoints were written by the version you just removed. Deserialisation fails, or worse succeeds into a step the old graph no longer has.
Containment: stamp the version tuple inside every checkpoint and make the resume path refuse a version it did not write. Then pick a drain policy in advance — usually "in-flight runs finish on the old version, new runs start on the new one" — and hold both versions deployable for the length of your longest run.
Trap: the bad version poisoned durable memory
v7 ran for six hours writing extracted preferences and summaries. Rolling back the code does not unwrite them, so v6 now reasons over v7’s mistakes — and a wrong "customer prefers no email contact" survives every deploy you make afterwards.
Containment: record write provenance on every durable memory record — agent version, run id, timestamp — so you can quarantine or purge by version. Keep extracted long-term insights TTL’d or re-derivable rather than treated as ground truth, and gate memory writes that will drive money-moving decisions.
Trap: the rollback target has quietly rotted
The version you plan to roll back to has not run for two months. In the meantime its pinned model snapshot was retired, an upstream tool changed its schema, and a credential rotated. Your rollback plan is a folder of assumptions.
Containment: run a rollback drill on a schedule — deploy the previous version to staging, run the golden suite, confirm the numbers. A rollback path that is not exercised is not a rollback path. Cheap to automate, and it converts the worst hour of an incident into a routine step.
Trap: partial rollback across two sources of truth
Your repository owns the container and the framework version; the platform console owns the instructions, the model selection, and the tool list — and someone edited them in the portal on Friday. You roll back the image and get v6 code driving v7 instructions. Nothing in either system is wrong; the pairing is new and untested.
Containment: one source of truth. If the platform owns agent configuration, define it as code and publish it through CI, so the platform’s version and your repository’s version move together — and forbid console edits in production.
Decision: roll back or roll forward?
Pre-decide, because you will not reason well at 02:00. Roll back when the new version clearly caused it, the previous version is known-good and drilled, and no state barrier stands in the way. Roll forward when persisted state has already diverged, when the previous version is also broken, or when the fix is small and testable — and remember that "roll forward" still means a full version bump through the gate, not a hot edit in a console.
Either way, the first action is neither: it is stopping the bleeding by halting new runs. That reflex — and the kill switch behind it — is the subject of the rollout module next door.
Containment sequence for a bad version
- Bad version detected in production
From a canary alert, a metric break, a support spike, or a human noticing. The version stamp on traces is what makes attribution possible.
- Stop new runs on it
Flag off, or point the endpoint back at the previous active version. Stop the bleeding before you diagnose.
- Runs already in flight?
- Finish them on the bad version, or abort and mark for replay
Never resume a checkpoint on a version that did not write it. Aborted runs go on a list — replay is a deliberate act, not an accident.
- Did it write state the previous version cannot read?
Memory records, checkpoints, cached summaries, new fields. This single question decides back versus forward.
- Roll forward: fix on top, ship as the next version
Through the same eval gate. A hot edit in a console is how a one-version incident becomes a two-version mystery.
- Roll back the endpoint to the previous version
The version you drilled last week, whose suite numbers you already know.
- Compensate the side effects it committed
Reverse what can be reversed, notify what cannot, and quarantine memory records written by that version id.
- Verify with the canary suite, then write the timeline
Add the failing case to the golden suite so this version can never ship again undetected.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.