Long-Context Economics

Lesson 4 of 4 in Context Windows and Long Context.

Long context is not a feature you switch on; it is three meters you spin faster.

Prefill compute → time to first token. The whole prompt passes through every layer before anything streams, so TTFT grows with prompt length — at least linearly, and steepening as Self-attention’s quadratic term gains weight at long lengths (Prefill compares every position with every other). A prompt ten times longer costs more than ten times the prefill.

KV memory → how many requests fit. Every token held in context keeps its keys and values resident in the KV cache for the life of the request — growth is exactly linear in tokens. On a fixed GPU, memory spent on one request’s long context is memory not available to batch other requests, so fleet Throughput drops and per-request cost rises even though the model never changed.

Per-step cache reads → speed of every subsequent token. Each Decode step attends over everything held so far, reading the whole cache from memory. A long context does not just cost you at the start — it adds a toll to every single output token (TPOT).

Line chart of KV cache size in gigabytes against context length from 4,096 to 131,072 tokens, on linear axes. Two straight lines through the origin region: the grouped-query attention configuration with 8 KV heads rises from 0.54 GB at 4k tokens to 17.2 GB at 131k tokens; the multi-head attention configuration with 32 KV heads rises four times faster, from 2.15 GB at 4k to 68.7 GB at 131k. The chart shows linear growth of cache memory with context length and the 4× gap between the two attention configurations.

KV-cache memory per request versus context length, computed from the standard formula: bytes per token = 2 (K and V) × layers × KV heads × head dim × bytes per value. Evaluated for a 7B-class shape (32 layers, head dim 128, FP16 = 2 bytes per value): with 8 grouped-query KV heads that is 131,072 bytes ≈ 0.13 MB per token; with full 32-head multi-head attention, 4× that. At a 131,072-token context the cache alone reaches ≈17 GB (GQA) or ≈69 GB (MHA) — for one request, before the model weights. Recompute for any shape in the KV Cache Calculator. (calculated — source: Formula per Pope et al. (2022) — Efficiently Scaling Transformer Inference)

Tool: KV Cache & VRAM Calculator — Plug in your own model shape, precision, batch size, and context length — the KV Cache Calculator turns this formula into your deployment’s actual memory line.

Two mitigations follow directly from the meters. First, Prefix caching: if many requests share a long prefix — the same system prompt, the same reference document — the serving stack can compute its KV entries once and reuse them, converting repeated prefill compute into a one-time cost (the mechanics live in The KV Cache). Second, send fewer tokens: if only a few pages of the filing cabinet matter, retrieving those pages usually beats shipping the cabinet on every call — when retrieval beats long-context stuffing is a decision the Adapting LLMs domain teaches in full.

The budgeting rule that survives all of it: budget by the tokens you actually make the model attend over, times how often you send them — not by the window size on the spec sheet. The advertised window is a ceiling. Your cost is the integral under your real traffic.

In production

Long-context tiers, premiums, and caching discounts exist on every platform because the costs in this lesson are physical: prefill FLOPs and resident cache bytes. Read each mechanism as cost recovery and the pricing stops looking arbitrary.

AWS

On Amazon Bedrock, input tokens are metered separately from output precisely because prefill is a real, length-dependent compute cost — and prompt caching discounts repeated prefixes because the platform genuinely skips that recompute. Treat the advertised window as a ceiling, not a plan: capacity math starts from your measured tokens-per-request distribution, and a template that quietly doubles input length doubles the metered side of every call.

Azure

Azure OpenAI in Azure AI Foundry enforces quota as tokens-per-minute, which makes long context a throughput decision, not just a cost one: a single window-filling request consumes quota that would have served dozens of short ones, and provisioned-throughput sizing rises with the token volume you actually push. Plan deployments from the context-length distribution of real traffic — the p95 request, not the maximum window, sets your capacity.

Google Cloud

Vertex AI meters generative usage in tokens and offers context caching for large prompt prefixes reused across calls — the managed form of the prefix-reuse economics above. If you instead self-host on GPU VMs, the KV formula in this lesson sets your memory floor: weights plus cache at your real context lengths and batch sizes decide which accelerator you need — size it with the GPU Sizing tool rather than the window printed on the model card.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.