Allocating the Window
Lesson 1 of 4 in Context Engineering at the Model Level.
Everything you learned about prompting in the last module ends up in one place: a single sequence of tokens, bounded by the model’s Context window. The window is not a container you fill — it is a budget you allocate, and Context engineering at the model level is the discipline of allocating it on purpose.
Five tenants compete for the space. Instructions — the System prompt and task rules. Examples — the Few-shot demonstrations that steer the model through In-context learning. History — prior conversation turns, the only tenant that grows on its own. Documents — the pasted or retrieved material that grounds answers in your facts rather than the model’s training data. And output headroom — the slice you must leave empty, because input and output share the same window: a prompt that consumes everything leaves the model no room to answer.
An unmanaged budget fails the way unmanaged budgets always fail: the growing tenant eats the others. Teams that append every conversation turn verbatim discover that by turn thirty the instructions are a rounding error in a sea of history, formatting rules stop being followed, and each request costs several times what it did at turn one. The fix is never “a bigger window” first — it is a policy per slice: instructions get a fixed reservation, examples earn their tokens or get cut, history gets truncated or summarized past a threshold, documents get a cap, and headroom is reserved up front.
Bar chart showing illustrative context-budget shares for two applications. Support copilot: instructions 12 percent, few-shot examples 8 percent, conversation history 45 percent, retrieved snippets 15 percent, output headroom 20 percent. Contract analyzer: instructions 6 percent, examples 2 percent, document under review 72 percent, history 2 percent, output headroom 18 percent.
Why be stingy at all, when windows keep growing? Because every included token pays rent, in three currencies at once. In money: input tokens are metered per request, and whatever your assembly code includes is re-sent — and re-billed — on every call. In latency: Prefill must process every input token before the first output token appears, so a fatter prompt directly delays time to first token. In attention: the model’s Attention weighs every token against every other, and material that is irrelevant to the current question still participates — diluting the signal you actually care about.
The rent is due whether or not the tenant contributes. A stale document, a redundant example, a pretty-printed JSON blob with three spaces of indentation — the model processes and you pay for all of it, even if the answer never touches it. That is the mindset shift of this module: not “what could help?” but “what justifies its rent?”
Key terms: Context engineering, Context window, In-context learning, Grounding, Prefix caching
The rent, itemized
Prefill compute. Processing the prompt is a forward pass over all n input tokens, and self-attention’s pairwise comparisons grow with roughly n² — the quadratic tax from the transformer domain. Doubling the material in your prompt more than doubles the attention work done before the first output token.
Per-step decode drag. The tax does not end at prefill. Each generated token attends over the entire cached context, so a bloated prompt makes every output token slightly more expensive to produce — reading a larger KV cache from memory at each step. A long prompt slows the whole answer, not just its start.
Memory. Every context token holds keys and values in the KV cache for the duration of the request. Cache memory is the scarce resource that limits how many requests a serving engine can batch together, so oversized prompts reduce a deployment’s effective capacity — a cost you pay even self-hosting, where there is no per-token invoice.
Billing. Managed APIs meter input and output tokens separately, and input includes everything your template resends per call. Prefix caching (next lesson) can discount the repeated part — but only if you lay the prompt out so the repeated part is actually a prefix.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.