Caching: prompt, semantic, and edge
Lesson 3 of 3 in GenAI Cost Engineering.
The cheapest token is the one never generated — and the second cheapest is the one never re-read. There are three caching layers, and they sit at three different depths of the request path. Edge caching (CloudFront) returns a stored response to a byte-identical public request before it touches your application. Semantic caching (ElastiCache or another vector-capable store) recognizes that a paraphrased question was already answered and returns the stored answer with zero model invocation. Prompt caching (inside Bedrock) cannot skip generation at all — it reuses the processed static prefix of the prompt so the model stops re-reading the same 8,000 tokens of system rules on every call. Three depths, three different savings, three different risks.
| Prompt caching (in Bedrock) | Semantic caching (you build it) | Edge caching (CloudFront) | |
|---|---|---|---|
What counts as a hit | The same static prefix, verbatim, within the TTL (which resets on each hit) | Query embedding within the similarity threshold of a cached query | Deterministic request hash matches — byte-identical request |
What it saves | Re-processing the prefix: discounted cached-read tokens and faster time-to-first-token — generation still runs | The entire invocation: zero tokens, near-instant answer | The entire backend round trip, served from the nearest edge location |
The failure mode | A variable injected at the top of the prompt breaks every hit — order the prompt static-first | A wrong match above threshold returns a confidently wrong answer; stale entries outlive the truth | A personalized or private response cached publicly leaks it to the next requester |
The exam tell | "The first 8,000 tokens of every request are identical" | "Thousands of users ask the same questions in different words" | "Identical public requests from a global audience" |
Key terms: Prompt caching (Bedrock), Semantic caching, Edge caching
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.