Checkpointing and replay: debugging runs you cannot reproduce
Lesson 5 of 5 in Deploying and Versioning Agents: Ship It Like Software.
Checkpoints earn their keep twice. The first payoff is operational: a run that crashes resumes instead of restarting. The second is the one that changes how you work — a checkpointed run is a run you can re-drive.
This matters because non-determinism broke the debugging loop you grew up with. "Reproduce the bug, then fix it" assumes the failure will happen again on demand. An agent may simply not take that path a second time. So you stop relying on reproduction and start relying on recording: a full trace of what happened, plus enough state to stand the run back up at the step before it went wrong.
Which is why the state-machine framing keeps paying off. Explicit steps give you natural checkpoint boundaries; a free-running loop gives you one boundary — after each tool result — and you should take it.
A checkpoint is not a memory dump. It is the minimum needed to continue: the version tuple, the step or node the run is at, the message history (or a reference to it), tool results so far, loop counters and budgets, pending approvals, and the idempotency keys of side effects already committed — so a resumed run does not send the email twice.
Note that last item. Without a record of what has already been done, resume is indistinguishable from redo.
In the checkpoint: the version tuple
The agent version id, the pinned model, the prompt hash, the tool schema hashes. Two reasons: the resume path can refuse to continue a run it did not write, and six weeks later you can tell which version produced the trace you are staring at.
In the checkpoint: position, budgets, and pending work
Current step or node, turn count against the budget, wall-clock spent, tokens spent, and any outstanding approval with its request id. Budgets must survive resume — otherwise every crash quietly refills the tank and a runaway run never hits its stopping condition.
In the checkpoint: side-effect ledger
An append-only list of committed effects with their idempotency keys: refund:order-8891 issued, email:ticket-4412 sent. On resume the agent consults the ledger before acting. This is the difference between at-least-once delivery and at-least-once side effects, and it is entirely your job — no framework knows that your refund endpoint is not idempotent.
Not in the checkpoint: secrets and raw sensitive payloads
Checkpoints are durable copies of everything the agent saw, which makes them a data-protection surface with a retention policy, an access policy, and a redaction step. Store credential references, never credentials; redact or tokenise sensitive fields; and set a retention window on purpose.
Platforms treat this the same way: Microsoft documents that capturing chat message content in Foundry traces is opt-in, precisely because message content may contain personal data. Your checkpoint store deserves the same caution — plus the reminder that anything an agent read can be attacker-supplied, so replaying a checkpoint means replaying hostile content too. Re-drive in a sandbox.
Not in the checkpoint: anything you can cheaply re-derive
Full document bodies, large tool payloads, model logits, the entire retrieval corpus. Store references and hashes. Checkpoint size drives resume latency and storage cost, and a checkpoint that takes twelve seconds to write will be silently disabled by the first engineer chasing p95.
AWS — as of 2026-09
AgentCore Runtime documents two substrates. microVM sessions are keyed by runtimeSessionId with isolated CPU, memory, and filesystem, terminate after 15 minutes of inactivity or an 8-hour maximum lifetime, and are explicitly described as ephemeral — durable context belongs in AgentCore Memory (short-term interaction events plus long-term extracted records). Runtime Instances (GA August 2026) run up to 14 days with persistent volumes that survive stops and re-attach on resume under the same session id.
Runtime also exposes InvokeAgentRuntime for agent reasoning and InvokeAgentRuntimeCommand for deterministic shell commands against the same session — useful when re-driving a run needs an exact, non-model action. (AgentCore Developer Guide — how Runtime works; Runtime Instances; Memory. Verify quotas and lifetimes in current docs.)
Azure — as of 2026-09
Foundry hosted agents get a per-session VM-isolated sandbox with a persistent filesystem ($HOME and /files), a configurable idle timeout of 2–60 minutes (default 15), scale-to-zero with stateful resume, and permanent session deletion after 30 days of inactivity. Alongside it sits a durable key-value state store of keyed JSON items that persists independently of compute — surviving crashes, restarts, and idle eviction — with a default 30-day item idle window that writes renew, and which Microsoft documents as able to hold framework checkpoints for LangGraph or Microsoft Agent Framework.
Conversation history is separate again: conversations are a durable server-side record of items — messages, tool calls, tool outputs. Three different lifetimes for three different kinds of state, which is the point. (Microsoft Learn — What are hosted agents?; Build with agents, conversations, and responses.)
Roll your own
No platform required, and worth understanding even if you use one. A step table (run_id, step_no, status, input_hash, output, version) plus an append-only side-effect ledger gets you resume, replay, and audit in one schema you fully control.
Framework checkpointers (the graph-style ones in the orchestration frameworks) give you this for free inside their model — with two conditions: the checkpoint format is theirs, so a framework upgrade is a state-format migration, and the durable backing store is still yours to choose, secure, and retain. Whatever you build, keep the version tuple in the row.
| Mode | What it answers | What it costs | The trap |
|---|---|---|---|
Trace replay — read the recording, execute nothing | "What did it do, and what did it see when it decided?" Ninety per cent of debugging ends here. | Nothing but storage. Needs spans with full inputs and outputs, not just timings. | Traces that log tool names without arguments and results. You will be able to see the crime and not the motive. |
Deterministic re-drive — resume from step k with recorded tool outputs | "Would a different prompt, model, or parameter have chosen differently at that exact step?" The sharpest tool for evaluating a candidate version against a real failure. | Model calls only. Fast enough to run across hundreds of past failures in CI. | Recorded outputs freeze the world. If the real bug was a changed API response, the replay will never show it. |
Live re-execution — resume from step k against real tools | "Does the fix actually complete the task end to end?" | Full run cost, plus real side effects unless the tools are dry-run or sandboxed. | Re-committing effects the original run already committed. This is what the side-effect ledger and idempotency keys are for. |
Counterfactual branch — fork at step k, change one tuple element | "Which single change fixes this class of failure?" One variable at a time, same starting state. | Model cost per branch, plus the discipline to change exactly one thing. | Comparing branches far past the fork: two runs diverge quickly, so judge the decision at the fork, not the prose forty turns later. |
Tool: Cloud Deployment Wizard — Walk a workload through topology, versioning, rollout, and rollback decisions — and see what your choices imply operationally — in the Deployment Wizard.
Interactive flashcard deck.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.