What earns a slot: the context budget

Lesson 1 of 5 in Memory Architectures: Engineering the Context at Scale.

From the foundations module you know the mechanics: the model is stateless, and everything it “remembers” is text someone chose to re-send in the context window. At production scale, that choice stops being an implementation detail and becomes the architecture problem — because the window is finite, every token in it costs money and latency on every call, and the model’s attention is a scarcer resource than the window itself.

So treat the context as a budget, not a bucket. Five claimants compete for it on every single turn:

the system prompt — the agent’s standing job description and tool guidance,
the task — the user’s actual request and its constraints,
the conversation history — everything said and done so far this run,
tool results — the fresh observations the loop just produced,
retrieved memory — whatever your stores and RAG pipeline injected.

The budget is zero-sum. Every retrieved chunk you inject is history you may have to compact; every verbose tool result crowds the instructions. Context engineering is the discipline of allocating that budget deliberately — deciding, for every candidate token, whether it earns its slot on this turn.

Five claimants on one budget — what each buys, and how each fails
Claimant	What it buys	When starved	When bloated
System prompt	Standing instructions, tool guidance, output contract — the rules of every turn.	The agent forgets its job mid-run: invents policy, misuses tools, stops escalating.	Instructions dilute each other; a 6,000-token rulebook is followed less reliably than a 600-token one.
Task / user request	The goal and its constraints — what “done” means for this run.	The agent solves an adjacent problem, or the original problem after the user amended it.	Rarely the culprit — but pasted-in requirements dumps belong in retrieval, not the task slot.
Conversation history	Continuity: decisions made, approaches tried, corrections issued.	The agent repeats work, re-asks answered questions, retries known dead ends.	Noise accumulates and old errors keep steering — this is where context rot lives (lesson 3).
Tool results	Fresh ground truth — what the world looks like now.	The model reasons from stale beliefs instead of current observations.	One 50,000-token log dump crowds out the instructions and the plan in a single turn.
Retrieved memory	Knowledge from long-term stores: preferences, facts, past episodes.	The agent re-derives known facts and forgets the user between sessions.	Marginally relevant chunks push out task-critical detail — retrieval volume is not retrieval quality.

Budget allocation becomes tractable once you see that every piece of information in an agent’s working set has exactly four possible dispositions:

Keep in context — task-critical, needed verbatim for the next few decisions. The failing test’s exact error message. The pinned architectural decision that constrains every edit.
Summarize — needed for continuity, not for its exact words. Forty turns of exploration collapse into a paragraph of conclusions (lesson 2).
Store and retrieve — durable value, needed occasionally. Write it to an external store; fetch it when — and only when — it’s relevant (lesson 4).
Drop — noise. Duplicated reads, raw dumps already mined for their one useful line, metadata nobody will consult. Deleting it is the cheapest optimization in the entire stack.

Strong context engineering is just applying the right disposition, early and repeatedly. Sort the real artifacts below.

Interactive sorting exercise: A coding agent is 50 turns into a long refactor. Allocate each artifact: keep it in context, summarize it, move it to a store for retrieval, or drop it.

Key terms: context engineering, context window, token, system prompt, long-term memory, context rot

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.