The Token Bill That Tripled
Traffic flat, invoice 3×: a post-mortem of four context leaks — unbounded history, duplicated chunks, an uncacheable preamble, and output nobody capped.
A composite teaching case: realistic fiction assembled from well-documented public patterns — not a real engagement.
We ran a customer-support assistant for a B2B software product: a chat UI, a managed model called through Amazon Bedrock, and RAG over a few thousand pages of product documentation. Every user turn assembled a prompt — System prompt, tool definitions, few-shot examples, the conversation so far, freshly retrieved doc chunks, the question — and streamed back an answer. Bedrock meters the two sides of a request separately, because they are different work: input tokens buy the parallel Prefill pass over the prompt, output tokens buy the serial Decode loop that writes the answer, and output is priced at a multiple of the input rate. We knew this the way you know a fact from onboarding. At launch the invoice was small enough that nobody read it closely, and the app was "done" — the prompt template kept changing, but prompt changes never went through the review that infrastructure changes did. That grace period is the villain of this story.
Wrong turn one: blame traffic. A bill that grows with flat traffic feels impossible, so we assumed the traffic numbers were lying. We spent a week hunting for hidden requests — a retry storm, a scraping bot, an integration test left running against production. The request logs were clean. Retries would have shown up as requests; nothing did. The first factor of the bill really was flat. Wrong turn two: blame the model. Next we argued the provider must have repriced. The price sheet said otherwise. Then we drafted a plan to switch to a cheaper model, and the counterfactual is worth writing down: a model at half the per-token rate would have halved the bill exactly once, and the growth curve would have kept climbing from the lower base — on its trajectory, it would have crossed the old bill again within two quarters. We were preparing to change p, the price per token, when the factor that had actually moved was the number of tokens. Neither wrong turn was stupid; both blamed a variable that had not moved, because those were the variables we had dashboards for.
Reading the meter correctly takes one idea from Prefill and Decode: input tokens are re-paid on every request. A token in the conversation history does not cost once — it rides along in the prompt and buys a slice of prefill on every subsequent turn. A conversation that re-sends its whole transcript pays an input bill that grows with the square of its turn count. Output has the opposite shape: each output token is bought once, but in the column priced at a multiple. With that lens we decomposed a sample of production prompts slice by slice, and found four leaks. All four were ours.
- History without a policy. The assembly code appended every past turn verbatim, forever. Average conversations had stretched from four turns at launch to twelve as users learned to go deeper — same request count, much longer transcripts riding along in each one.
- Duplicated retrieval. The retriever ran fresh each turn and its chunks were being written into the stored transcript. By turn ten, the same passages sat in the prompt three and four times — billed every time, and competing with each other in Attention.
- An uncacheable preamble. We had turned on Bedrock prompt caching months earlier and considered the box ticked. A later template refactor moved a session header — user id, timestamp — above the system prompt. Prefix matching is exact from token zero, so the hit rate quietly went to zero, and our 3,200-token preamble (grown from 900 by a year of accreted instructions) paid full-rate prefill on every call.
- Unbounded output. No
max_tokens, plus a well-meaning "be thorough" instruction from a quality push. Answers had swollen to restate the retrieved passages back at the user — verbose text nobody asked for, in the expensive column.
Where one request’s tokens came from (before the fix)
- User sends turn n
The question itself: ~80 tokens. Everything else below is packaging we chose.
- Session header rendered first
User id and timestamp at the top of the prompt — different on every request. Leak 3: one differing token at position zero forfeits every cache match behind it.
- Preamble appended (~3,200 tokens)
System prompt, tool definitions, few-shot examples. Stable and cacheable in principle — but sitting behind the header, never matched.
- Full transcript appended
Every past turn verbatim, including past retrieved chunks — growing ~2,400 tokens per turn with no ceiling. Leak 1.
- Top-5 chunks retrieved fresh (~2,000 tokens)
Appended below the transcript that already contains earlier copies of the same passages. Leak 2: the same paragraph, billed several times per request.
- Prefix match from token zero?
Prompt caching compares exact tokens from position zero. The per-request header guarantees a miss before the comparison reaches the preamble.
- Full-rate prefill over everything
Every token above — preamble, transcript, duplicates — re-paid at the fresh-input rate, this turn and every turn after it.
- Decode with no cap
No max_tokens, "be thorough" in the instructions: the answer restates the chunks. Leak 4, in the column priced at a multiple.
Line chart of input tokens per request against conversation turn number, turns 1 to 20. The before-fix line starts at 5,280 tokens at turn 1 and climbs steadily to 51,260 at turn 20 — unbounded linear growth per request. The after-fix line starts at 4,480, rises gently to 7,300 by turn 8, and stays flat at 7,300 through turn 20 because history is windowed and summarized. The gap between the lines widens every turn.
| Leak | Why the meter grew | The fix |
|---|---|---|
History never truncated | Every retained token re-buys Prefill on every later turn — resend-everything makes a conversation’s total input bill quadratic in its turn count | A history policy: last six turns verbatim, older turns rolled into a ~300-token summary, hard cap on the slice — the budget discipline from Context Engineering |
Retrieved chunks duplicated | Chunks were stored into the transcript and re-retrieved fresh — the same passage billed three or four times in one request, and again next turn | Chunks render in their own slice outside stored history, deduplicated by chunk id, capped at top-3; the transcript keeps chunk references, not chunk text |
Preamble re-sent uncached | A per-request session header above the System prompt broke exact prefix matching from token zero — Prefix caching hit rate was zero, so 3,200 stable tokens paid the fresh-input rate every call | Stable content first and byte-identical — system prompt, tools, examples — with a Bedrock cache checkpoint after it; all per-request fields moved to the tail (The KV Cache) |
Output unbounded | No |
|
The code changes took ten days. The structural change mattered more: prompt assembly stopped being string concatenation and became a context budget — named slices (preamble, history, retrieval, question, output headroom), each with an owner, a cap, and a policy for what happens at the cap. That is the discipline taught in Context Engineering, and it turned "the prompt" from a shared scratchpad into a system with invariants.
The cache fix was pure layout. Everything stable moved to the front and became byte-identical across requests; the cache checkpoint sits right after it; everything per-request — session metadata, history, chunks, question — comes behind. Hit rate went from zero to nearly every call after a conversation’s first, which repriced the preamble from the fresh-input rate to the cached-input rate and handed us a TTFT improvement for free, since a matched prefix skips its share of prefill. The mechanism — and why providers meter cached input differently at all — is the cache-reuse lesson in The KV Cache.
Output caps were the change everyone feared and nobody noticed. We sized max_tokens per route from the length distribution of answers users actually rated as helpful, tightened the format instructions, and told the model to cite chunk ids instead of quoting passages. Support satisfaction scores did not move. The cheapest, fastest token — as Prefill and Decode puts it — is the one you never generate.
Within one billing cycle the model line settled at a bit under half of its peak — not back to launch levels, because the product genuinely did more than at launch, but back on a curve that tracked usage instead of outrunning it. The diagnosis also explained two "unrelated" complaints we had been shrugging at: long conversations had been getting slower (every turn re-prefilled a swelling transcript) and worse at following format rules (the instructions were a rounding error in a sea of history and duplicate chunks — the failure mode Context Engineering warns about). What we actually kept from the incident is instrumentation and process. Tokens per request, by slice, sits on the same dashboard as request count; cache hit rate is an alertable metric with a floor; and any pull request that touches a prompt template must state its expected token delta, the same way a schema migration states its lock time. The bill has not surprised us since — not because it cannot grow, but because it can no longer grow silently.