Prompt Caching by Design

Cache-hit economics are decided the day the template is written: stable content first, variable content last, one versioned prefix per deploy — and named variables for knowing it is working.

Prompt caching is the rare optimization that is decided at design time and merely collected at runtime. The mechanism — Prefix caching on the serving engine's KV cache — reuses the computed state of a prompt's opening tokens whenever a new request begins with the exact same sequence, skipping that share of Prefill. All three clouds meter the reused portion as cached input, on better terms than fresh input, because the work genuinely was not done twice. The mechanism end to end lives in The KV Cache; this note is about the habit that decides whether you ever collect on it. What the pricing page does not say is that the discount is structural. Matching is exact and prefix-only, starting at token zero: the engine can only reuse the part of the prompt that is byte-identical across requests and sits in front of everything that varies. A template fixes its own ceiling on cache hits the day it is written. Retrofitting the order later means touching a prompt that is already passing your regression suites; getting it right on day one costs nothing. So we treat cache layout the way we treat database indexes — a day-one design decision with a known access pattern — and this is the pattern: a layout law, a versioning discipline, and the arithmetic that tells you whether it is paying.

Why the meter moves: a fresh prompt Token buys a slice of Prefill — every layer, every attention comparison, its keys and values written into cache blocks. A token covered by a matched prefix buys almost none of that. Its state is already resident; the engine's work shrinks to matching the prefix and pointing the request's block table at blocks that already exist (PagedAttention made that pointing cheap). TTFT drops because most of the prefill is skipped, and the GPU compute that prefill would have burned is freed to serve other requests, which is why providers can price cached input below fresh input and still come out ahead. The discount is a mechanism, not a promotion — and a template that never matches simply re-buys the same prefill on every call. The same law governs multi-turn and agent traffic, where it is worth the most. A conversation that only ever appends — new tool results and user turns at the end, history never rewritten — is a prefix that grows monotonically: turn twelve's prefill reuses turns one through eleven and pays only for what turn twelve added. Middleware that rewrites history breaks this silently: re-rendering timestamps on old turns, compacting earlier messages, re-serializing tool results in a different key order — each one converts every subsequent turn into a full prefill. Append-only is not just tidy Context engineering; it is the multi-turn form of the layout law.

The countermeasure is to stop letting application code assemble the stable head per request. Treat it as a build artifact: rendered once at deploy time, serialized deterministically, hashed, and given a version. What we actually maintain:

• One canonical rendering per version. The head is produced by the template build, not by string concatenation in the request path. Deterministic serializers for tool definitions — sorted keys, fixed number formatting — are part of the contract, and a unit test asserts the rendered head's hash so an accidental reordering fails CI instead of the cache.
• Few, long-lived versions. Ship head changes as deliberate cutovers, not gradual interleavings. A deploy is a planned fragmentation event: the hit rate dips while traffic moves to the new prefix and recovers within the cache's lifetime. A dip that does not recover means two versions are still interleaving — that is a bug, and the hit-rate chart is where it shows.
• One byte-stable head per experiment arm. Experiments on prompt wording are legitimate; randomizing inside the head is not. Two arms means exactly two prefixes, each stable, each earning its own hits.
• A registry, not tribal knowledge. Version → hash of the rendered head, plus the version stamped on every request log next to the provider's cached-token count. That one join answers "which template change moved the hit rate" without archaeology. Per-request personalization still goes in the tail. The only per-tenant heads we keep are for tenants whose traffic alone sustains the reuse frequency — measured, not assumed.

# ================= STABLE HEAD — rendered at deploy, byte-identical =================
# template version: {{TEMPLATE_VERSION}}   <- changes only on deploy, never per request

You are {{ASSISTANT_ROLE}}, the assistant for {{PRODUCT_NAME}}.

## Rules
{{POLICY_BLOCK}}

## Output contract
{{OUTPUT_CONTRACT}}

## Tools
{{TOOL_DEFINITIONS_JSON}}
# canonical serialization: sorted keys, fixed formatting — enforced by a hash test in CI

## Worked examples
{{FEW_SHOT_EXAMPLES}}
# fixed set, fixed order — never sampled per request

# ================= VARIABLE TAIL — rendered per request, placed last =================

## Session
Date: {{TODAY}}
User: {{USER_PROFILE}}

## Retrieved context
{{RETRIEVED_CHUNKS}}

## Request
{{USER_QUERY}}

Name the variables and the economics stop being vibes. Per request:

• S — stable-head length in tokens; V — average variable-tail length. Input per request is T = S + V.
• H — the measured hit fraction: cached input tokens ÷ total input tokens, read straight from the usage fields the provider returns on every response. On single-shot traffic the layout sets the ceiling — H ≤ S / (S + V) — while append-only conversations can run higher, because every earlier turn joins the reusable prefix. Operations decide how close you run to the ceiling: how many head versions are live, whether traffic arrives within the cache's lifetime, whether anything upstream is fragmenting the prefix.
• d — the cached-price ratio for your provider and model: price per cached input token ÷ price per fresh input token. Effective input spend per request is then T × p_fresh × (1 − H × (1 − d)): the fraction of the input bill the cache removes is H × (1 − d). Both factors matter — a high hit rate against a weak discount saves little, and a strong discount a fragmented template never triggers saves nothing. For explicit caches that bill residency (a rate per token per unit time for keeping the block live), add that term and the break-even falls out directly: the cache pays when hits during the cache's lifetime × per-hit prefill saving exceed the residency cost — which is why a corpus queried constantly caches well and one queried hourly may not. Two dashboards close the loop. H by template version shows adoption after each deploy and catches the dip-that-never-recovers. And one alarm: H falling with no deploy in the window is template drift — something in the request path started varying the head, and the cached-token counter noticed before anyone read a diff. In practice that alarm catches prompt-assembly bugs days before the invoice would. Baseline fleet arithmetic for the fresh-token side lives in Cost Modeling; for intuition about what the provider is holding in memory on your behalf, put your model shape into the KV Cache Calculator.