The three horizons of memory
Lesson 1 of 5 in Agent Memory: Remembering and Forgetting on Purpose.
Start from the fact that makes everything else necessary: the model has no memory at all. An LLM is a pure function from context to output. The model that answers your fifth message has no recollection of the first four — the application re-sends the entire conversation with every request, and the model reads it fresh each time, like a goldfish handed a diary.
So when an agent "remembers", that is never a property of the model. It is an engineering artifact bolted on around it. Every memory system, however elaborate, answers exactly one question: which text gets placed back into the context window, and when? Memory is context engineering with a persistence layer.
Sort every memory mechanism by one variable — how long it lives — and the whole landscape resolves into three horizons:
- Scratchpad — memory within a single run. The plan the agent drafted, the tool call results piling up as the agent loop iterates, the notes-to-self between steps. It lives in the context (or a temp file) and dies when the run ends.
- Session — memory within a conversation. The message history the application re-sends turn after turn, plus any compacted summary that replaces old turns when the context fills up. It dies when the conversation ends.
- Long-term — memory across sessions. Anything deliberately written to storage — a file, a database, a vector index — so a future session can read it. It dies only when someone deletes it. That "only when someone deletes it" is where most of this module’s trouble lives.
Key terms: memory, scratchpad, session memory, long-term memory, context window, context engineering
| Horizon | Lifetime | Where it lives | How it reaches the model | Typical contents | Characteristic risks |
|---|---|---|---|---|---|
Scratchpad | One run of the agent loop | The context itself; sometimes a temp file | Already in context — accumulated turn by turn | Plans, tool outputs, intermediate reasoning | Context overflow mid-task; the agent losing the plot on long runs |
Session | One conversation | App-side message history (and compaction summaries) | Re-sent in full — or summarized — every request | Everything said so far; earlier decisions in this chat | Compaction silently dropping the detail that mattered; token cost growing every turn |
Long-term | Until explicitly deleted | Files, databases, vector stores — outside any conversation | Loaded at session start or retrieved on demand (RAG-style) | Preferences, project facts, learned procedures | Stale facts, poisoning, privacy accumulation — lessons 3 and 4 |
Interactive sorting exercise: Which horizon does each of these live on?
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.