Compaction: checkpoint, then summarize
Lesson 2 of 5 in Memory Architectures: Engineering the Context at Scale.
Long runs hit a wall — two walls, actually. The hard wall is the context window limit: one more turn simply doesn’t fit. The soft wall arrives earlier: cost and quality degrade as the history bloats (lesson 3). Either way, at some point the history must shrink, and the standard move is compaction: replace the older portion of the conversation with a model-written summary and keep only the recent turns verbatim.
Compaction is lossy by construction. A summary is a model’s opinion about what mattered, written before knowing what the future will need. The summarizer cannot tell which exact string, which abandoned approach, which offhand user correction will turn out to be load-bearing three hundred turns from now. That asymmetry — compress now, discover the loss later — is why naive compaction quietly breaks long-running agents.
The production answer is an ordering rule: checkpoint first, summarize second. Before the history shrinks, everything durable — decisions made, open questions, the task list, exact identifiers, known dead ends — gets written outside the context, verbatim, to a file or store. Only then does the summarizer run. The summary provides continuity; the checkpoint preserves truth.
Checkpoint-then-summarize
- Agent loop running
- Context over threshold?
Trigger on a token threshold (commonly ~70–85% of the window), a turn count, or proactively before ingesting a known-large tool result — not only at the hard limit.
- Checkpoint durable state externally
Write decisions, open tasks, exact identifiers, and dead ends to a file or store — verbatim, dated, outside the context. This copy is the recovery point, and the ops discipline behind resumability.
- Summarize the older turns
A structured summary template beats freeform: goal, decisions so far, current state, do-not-retry list, next steps.
- Replace old turns with the summary
The system prompt stays; recent turns stay verbatim; the middle of the history becomes one compact block.
- Reassemble: prompt + summary + recent turns + checkpoint pointer
The agent knows where the verbatim truth lives — it can re-read the checkpoint when the summary proves too thin.
- Continue the run
Loss 1 · Exact strings — identifiers, paths, error codes
The summary says “fixed a configuration issue”; it no longer says which key, in which file, changed from what to what. Exact strings are the first casualty of summarization because they look like detail and compress like detail — but agents act on exact strings.
Countermeasure: the checkpoint carries the verbatim values; the summary carries a pointer to the checkpoint.
Loss 2 · Negative results — the dead ends
“We tried the streaming API and it fails on payloads over 1 MB” is among the most valuable facts a long run produces — and summarizers reliably cut it, because a thing that didn’t work reads as irrelevant to a narrative of progress. The agent then rediscovers the dead end at full price, sometimes repeatedly.
Countermeasure: a structured summary template with a mandatory do-not-retry section. If the section is empty, the summarizer must say so explicitly.
Loss 3 · Corrections — the user’s “stop doing that”
Mid-run corrections live in the history as ordinary turns: “never push directly to main.” A summary softens never into prefers not to — or drops it entirely — and the guardrail dies in the compression. The user experiences an agent that was told, agreed, and then “forgot.”
Countermeasure: corrections are instruction-tier content, not history-tier. Promote them into the pinned instructions block that compaction never touches.
Loss 4 · The audit trail — why the agent believes what it believes
After compaction, the working context can no longer explain itself: the summary asserts conclusions whose supporting turns are gone. For debugging, evals, and incident response, that provenance matters enormously.
Countermeasure: compaction should only ever destroy the context copy. The full raw transcript belongs in an episodic log (lesson 4) — the run remains reconstructible even when the working set is not.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.