When Context Isn’t Enough

Lesson 4 of 4 in Context Engineering at the Model Level.

The stuff-the-docs baseline fails in exactly three ways, and naming them precisely matters because each wall points at a different piece of retrieval machinery.

The scale wall: the corpus dwarfs the window. Ten thousand support tickets, a decade of contracts, a whole codebase — no window holds it, and no amount of prompt discipline changes the arithmetic. Long-context models move the wall; they do not remove it, and lesson one’s rent still accrues on every token you manage to squeeze in.

The freshness wall: updates outpace prompts. Context is a snapshot taken when the prompt was assembled. If the material changes weekly and your paste is rebuilt monthly, the model answers confidently from a stale world — a worse failure than refusing, because nothing looks wrong. When you cannot re-fetch the current version at assembly time, the facts have to live somewhere that is queried live, not copied forward.

The selection wall: you don’t know what’s relevant until the question arrives. This is the subtle one. Even a corpus that technically fits may serve you badly whole — the decisive paragraph competing with thousands of irrelevant ones for Attention, at full rent. And with a corpus that doesn’t fit, someone must choose what enters the window per question. A fixed choice cannot do it; the choice has to be computed from the question itself. That computation — find the pieces of a large corpus most relevant to this query, right now — is Retrieval, and RAG is the pattern of wiring it into the prompt.

Stay with context, or move to retrieval?

Interactive decision tree — outcomes:

  • Stay with context

    The baseline is working: full coverage, one failure layer, no index to operate. Keep the grounding contract, keep the stable-prefix layout, and keep the evaluation that told you it works — it becomes the benchmark any future pipeline must beat.

  • Scale wall → retrieval is mandatory

    No window fits the corpus, so something must choose per question what enters the prompt. That chooser is a retriever over an index. Your grounding instructions and citation contract carry over unchanged — only the source of the documents section changes.

  • Freshness wall → query a live index

    Copied-forward context answers from a stale snapshot without looking wrong. Move the material behind a query executed at request time, and make index update lag a monitored number — freshness becomes a pipeline property you can measure instead of an assumption.

  • Cost wall → retrieval as a cost lever

    Everything works, but you are paying rent on the whole corpus for every request. Retrieval sends slices instead — trading per-request token cost for the standing cost of embedding, indexing, and operating the pipeline. Do the arithmetic on your real traffic before committing; caching discounts may already have closed the gap.

  • Selection wall → retrieve, then consider reranking

    Relevance is per-question, so it must be computed per question. Retrieval narrows the corpus to candidates; if quality still suffers because near-misses crowd the window, Reranking sharpens the final cut. The next two modules open exactly this machinery.

Notice what the walls do not say: nothing here claims context engineering was a detour. The grounding contract, the budget discipline, and the stable-prefix layout all survive the move — RAG simply changes who fills the documents section, from a loader to a retriever. (And when a model itself decides mid-task whether and what to retrieve, you are orchestrating an agent — our sister AI Agent Academy teaches that pattern; here we go under it, into the retrieval machinery.) The mechanics begin in RAG Mechanics: Embeddings and Search: how text becomes vectors, how similarity search over millions of documents runs in milliseconds, and where it silently fails.

In production

Context layout is a cost lever on every managed platform, because all three clouds meter input tokens and all three ship mechanisms that reward stable, well-ordered prompts.

AWS

Amazon Bedrock offers prompt caching on supported models: repeated prompt prefixes are checkpointed at the serving layer, and requests that reuse a cached prefix skip recomputing it — billed at a discount and returning first tokens sooner. The mechanism reads your layout literally: only an identical leading span qualifies, so the stable-prefix-first discipline from lesson two is what converts a big system prompt from a per-request tax into a mostly-cached asset.

Azure

Azure OpenAI in Azure AI Foundry applies prompt caching automatically on supported models when requests share a sufficiently long identical prefix — same mechanism, same layout sensitivity. Quotas add a second lever: rate limits are denominated in tokens per minute, so a slimmed, well-allocated context does not just cost less per call, it raises how many requests fit under the same quota before throttling.

Google Cloud

Vertex AI exposes context caching as an explicit API object: you create a cache from large stable content (a corpus, a long system preamble) and reference it across requests instead of re-sending it — the stuff-the-docs baseline with the re-send cost engineered away. Where a model’s pricing is tiered by prompt size, total context length also decides which per-token rate a request lands in, making the budget from lesson one a literal line item.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.