Why Output Costs More Than Input

Lesson 2 of 3 in Prefill and Decode.

Look at almost any managed-LLM price sheet and the same asymmetry appears: an output token is priced above an input token, commonly at a multiple of the input rate. That is not marketing. It is the prefill/decode split expressed in currency.

Count the forward passes in last lesson’s toy request. The 1,000 input tokens were processed in one parallel pass — the weights were read from memory once, and the arithmetic for all thousand positions shared that read. The 200 output tokens needed 200 sequential passes, the full weight set re-read for every single one. Measured in raw FLOPs, a token is a token: the arithmetic per token is roughly the same on both sides. Measured in time the hardware is tied up — the thing a provider actually sells — an output token is dramatically more expensive, because its pass is serial, memory-starved, and unshared.

Bar chart comparing forward passes for one illustrative request. The input bar shows 1 pass covering all 1,000 prompt tokens; the output bar shows 200 passes, one per generated token.

The same toy request (1,000 tokens in, 200 out) counted in forward passes through the full weight stack. All input tokens share one parallel prefill pass; every output token demands its own decode pass. Counts are illustrative for one request. Note what this chart does not say: FLOPs per token are similar on both sides — it is the serial, bandwidth-starved passes that make output time expensive. (illustrative — source: Cost framing follows Pope et al. (2022), arXiv:2211.05102)

This is why per-token pricing comes split in two. When a provider quotes separate input and output rates, the ratio reflects what each side does to their fleet: input tokens are prefill work — dense, parallel, batchable — while output tokens occupy the serial, bandwidth-bound decode loop, holding capacity that could serve someone else. The same asymmetry prices your Context window strategy: a huge prompt costs real money, but it costs it once per request, in the cheap column. The expensive column is the one the model writes.

Run your own traffic shape through the Token Cost Estimator and the split becomes visceral: workloads with identical total token counts can differ sharply in cost purely by which side of the ledger the tokens sit on. And the architecture decisions that set both rates in the first place — attention variants, MoE, context length — have their own module: Architecture in Production.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.