Structure and Placement

Lesson 2 of 4 in Context Engineering at the Model Level.

Once you know what goes in the window, the next question is where. Order matters twice — once to the serving stack, once to the model itself.

To the serving stack, order decides what can be reused. Prefix caching lets the engine skip recomputing the KV cache for the part of a prompt it has seen before — but the match runs from the first token forward and stops at the first difference. Everything after the first changed token recomputes at full price. The layout rule falls straight out of the mechanism: stable content first, volatile content last. System instructions, tool and format definitions, and few-shot examples — identical on every request — form the cacheable prefix. Retrieved documents change per request; history changes per turn; the user’s question changes every time. Each belongs after everything more stable than itself.

A cache-friendly prompt layout, top to bottom

  1. System instructions

    Role, rules, tone, refusal policy. Identical for every request — the anchor of the cacheable prefix. No timestamps, no user names, nothing volatile.

  2. Output format & tool definitions

    Response schema, section headers, tool signatures. Changes only when you deploy — still part of the stable prefix.

  3. Few-shot examples

    Worked demonstrations. Stable across requests, so they cache — one reason examples are cheaper than their token count suggests.

  4. Grounding documents

    Pasted or retrieved material. Changes per request — the cache boundary usually lands here. Delimit each document and give it a citable ID.

  5. Conversation history

    Prior turns, under the truncation or summarization policy from lesson one. Changes every turn.

  6. Current question + volatile values

    The user’s message, the timestamp, the request-specific facts. Most volatile, so it goes last — nearest the answer, costing nothing in cache terms.

To the model, order decides what gets used. Liu et al. (2023) — “Lost in the Middle” — moved a single relevant document through a stack of distractors and measured question-answering accuracy at each position. For the models they tested in mid-2023, accuracy traced a U-shape: strongest when the answer sat at the very start or very end of the context, weakest in the middle. That result is model- and task-specific — training recipes have targeted it since, and newer models often show flatter curves — but it earns a default: put the material the model must not miss at the edges, and never bury the decisive passage at position ten of twenty. The serving domain’s context-windows module turns this into a measurement you can run on your own model in an afternoon.

The last structural lever is sectioning. A prompt with labeled regions — delimited documents, a marked history block, an explicit question line — gives the model addressable structure: instructions like “answer using only the material inside the documents section” or “cite the document ID for every claim” need a there to point at. Delimiters (XML-style tags, markdown headings, fenced blocks) cost a handful of tokens and buy you instructions that reference regions instead of hoping. They also make prompts diffable and testable — which is what “prompts are code” meant in the previous module.

A production prompt skeleton: stable prefix first, volatile tail last
You are {{ROLE}}, assisting with {{TASK_DOMAIN}}.

Rules:
- Answer using only the material inside <documents>. If it does not contain the answer, say so explicitly.
- Cite the id of every document you rely on, like [doc-2].
- {{ADDITIONAL_STANDING_RULES}}

Output format:
{{OUTPUT_SCHEMA_OR_SECTION_HEADERS}}

Examples:
{{FEW_SHOT_EXAMPLES}}

<documents>
{{RETRIEVED_OR_PASTED_DOCUMENTS_EACH_WITH_AN_ID}}
</documents>

<history>
{{TRUNCATED_OR_SUMMARIZED_CONVERSATION_HISTORY}}
</history>

Current date: {{REQUEST_TIMESTAMP}}
User question: {{USER_QUESTION}}

Everything above is identical on every request — that is the cacheable prefix, so never let a timestamp, user name, or request ID creep into it. Volatile values sit at the very end, where they invalidate nothing. The delimited sections give the grounding and citation rules a concrete target to point at.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.