The Window Is a Budget

Lesson 1 of 4 in Context Windows and Long Context.

The Context window on a spec sheet reads like room to relax in. In production it behaves like a budget line. The window is the maximum number of tokens the model can attend across at once — and the prompt and the output share it. Every token the model generates occupies a position in the same window as the tokens you sent. A model with a 32k window and a 31,800-token prompt has about 200 tokens of room left to answer you, no matter what you set as the output limit.

What actually fills the window in a real application is mostly not the user. A typical retrieval-augmented assistant sends: a System prompt with instructions and guardrails, tool or function definitions, a few worked examples, the retrieved documents, the conversation so far — and, at the very end, the user’s actual question, often the smallest item on the invoice. Reserve room for the answer and the budget is spent before anyone typed anything.

Bar chart of seven budget items for one hypothetical 16,000-token request: system prompt and instructions 1,200 tokens; tool definitions 800; few-shot examples 1,500; retrieved documents 6,000; conversation history 4,000; the user’s current message 300; and 2,200 tokens reserved for the output. Retrieved documents and history dominate, while the user’s message is the smallest bar.

Where a 16k-token budget goes in a hypothetical retrieval-augmented assistant. Toy numbers, realistic shape: the machinery — instructions, tools, examples, retrieved context, history — dwarfs the user’s question, and the output reservation must come out of the same budget. Your own split will differ; the point is to draw it for your app. (illustrative — source: Liu et al. (2023) — multi-document QA prompts have exactly this anatomy)

Sooner or later a request exceeds the budget — a long conversation, a big document, a generous retriever — and something has to go. That something is a design decision, and each option loses a different thing. The dangerous part is that every option fails silently: the API call still succeeds, the answer still streams, and the missing content announces itself only later, as the model “forgetting” a constraint from turn one or missing the clause at the end of the contract.

The standard truncation strategies, and what each one silently loses. Most real systems combine several.
Strategy	How it works	What it silently loses
Drop oldest turns	Evict conversation history first-in, first-out until the request fits	Early commitments: the name the user gave in turn one, the constraint they set before the small talk. The model does not know the history was cut
Summarize history	Replace old turns with a model-written summary and keep recent turns verbatim	Whatever the summarizer judged unimportant — judged before knowing what turn 40 would need. Adds a summarization call’s cost and latency
Re-rank and trim retrieval	Keep only the top-scoring retrieved chunks; cut the tail	Whatever the ranker mis-scored. Answer quality now depends on retrieval quality — a dependency worth its own monitoring
Hard-truncate the document	Cut the input at the token limit, keeping the beginning	The ending — which is where conclusions, exception clauses, and signature blocks live. Often the worst possible choice for contracts and reports

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.