Prefix Caching and Reuse

Lesson 4 of 4 in The KV Cache.

So far the cache has lived and died with one request. The last move is to let it outlive the request — because for the same model at the same precision, a token’s keys and values depend only on the tokens at and before its position. Two requests that begin with the identical token sequence produce identical cache entries for that prefix. Computing them twice is pure waste. Prefix caching exploits this: the engine keeps prefill results around (with PagedAttention, conveniently, as blocks — hashed by content, shared between sequences, copied only on divergence). When a new request arrives, the engine finds the longest cached prefix that matches it exactly and runs Prefill only on the remainder. Production traffic is full of such prefixes: the shared System prompt in front of every request, tool definitions, few-shot examples, a long document your users keep asking questions about. The meters move immediately. TTFT drops because most of the prefill is skipped; the GPU compute that prefill would have burned is freed for other requests, so Throughput rises. What reuse costs instead is memory: cached prefixes occupy blocks while they wait to be matched, which is a bet that they will be matched soon.

Two requests, one shared prefix

  1. Request A arrives

    System prompt + tool definitions + few-shot examples + question 1. Nothing is cached yet.

  2. Full prefill for A

    One parallel pass computes keys and values for every prompt token, filling cache blocks.

  3. Prefix blocks kept, hashed by content

    The blocks holding the shared preamble stay in the pool after A completes, indexed so an identical prefix can find them.

  4. Request B arrives

    Same system prompt, tools, and examples — only the user question differs.

  5. Longest cached prefix match?

    Matching is exact and prefix-only, typically at block granularity: identical tokens, identical order, from position 0.

  6. Reuse blocks; prefill only the new tokens

    B’s block table points at the shared physical blocks. Compute is spent only on the question — TTFT drops accordingly.

  7. Full prefill from token 0

    One changed token early in the prompt invalidates everything after it.

  8. Decode streams the answer

    From here both paths behave identically; only the prefill bill differed.

This is also why providers meter cached input tokens differently from fresh ones. A fresh prompt token buys a slice of Prefill: every layer, every attention comparison. A token covered by a cached prefix buys almost none of that — its keys and values already sit in blocks in memory, and the engine’s work shrinks to matching the prefix and pointing a block table at it. Same token count, a fraction of the compute; the discount is a mechanism, not a promotion. What the provider still carries is the other half of the bet — holding those blocks resident while they wait to be matched.

The lever this hands you is prompt structure. Matching is exact and prefix-only — one differing token invalidates everything after it — so the shared, stable content must come first: the System prompt, tool definitions, few-shot examples, the reference document everyone keeps asking about. The per-request material — the user’s question, retrieved snippets, anything carrying a timestamp or session id — goes last, where a mismatch costs only itself. A template that stamps the current time above the system prompt forfeits every cache hit it would otherwise earn. Structuring agent sessions so that multi-turn tool loops stay on cached prefixes is its own craft — our sister AI Agent Academy’s territory.

In production

All three clouds’ managed model APIs expose this lesson as a caching mechanism on the token meter: a repeated prompt prefix skips its prefill, and the cached portion is metered on better terms than fresh input — a discount that only prompts with stable prefixes ever collect.

AWS

Amazon Bedrock’s prompt caching lets you mark cache checkpoints in a prompt: the computed state of everything before a checkpoint is stored and reused when a request arrives with a matching prefix, skipping that portion of prefill and metering it as cached input. The structural consequence is exactly this lesson’s rule — keep the system prompt, tool definitions, and shared examples byte-stable at the front of the template and let only the tail vary, or the checkpoints never match twice.

Azure

Azure OpenAI in Azure AI Foundry applies prompt caching to requests whose opening token runs match a recently seen prefix: the matched portion skips recompute and is metered as cached input, and cached routing can also improve time to first token. Because matching starts at token zero, ordering is the whole game — a template that injects per-user or per-session fields early quietly opts out of the mechanism, while a stable-first template collects the benefit on every call.

Google Cloud

Vertex AI offers context caching for exactly the shared-prefix shape: a large stable block — a corpus, a long system preamble — can be cached and referenced across calls, with the cached tokens metered on reduced terms and a separate charge for keeping the cache resident. That residency line makes this lesson’s trade visible on an invoice: prefix reuse converts repeated prefill compute into memory held between requests, and it pays only if the prefix really is shared and stable.

Key terms: KV cache, PagedAttention, Prefix caching, Prefill, Grouped-query attention (GQA)

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.