Causal Masking: No Peeking at the Future

Lesson 4 of 4 in Self-Attention: Every Token Looks at Every Token.

An Large language model (LLM) generates left to right: when it is choosing token 6, tokens 7 and beyond do not exist yet. But training runs on complete texts, where the future is sitting right there in the sequence. And for efficiency, training makes one forward pass do every prediction at once — position 1 predicts token 2, position 2 predicts token 3, and so on down the sequence, all in parallel. That is Next-token prediction at industrial scale, and it has an obvious cheat: if position 5 could attend to token 6, it could read the answer off the page. The model would learn to copy the future, not to predict it — and collapse the moment you asked it to generate, when there is no future to copy.

The fix is Causal masking: before the softmax, add negative infinity to every score where a token would look at a later position. Since e^−∞ = 0, those positions get exactly zero weight, and the softmax renormalizes so each row’s budget of 1 is spent entirely on the token itself and its past. The score matrix becomes lower-triangular — the upper-right triangle is dead by construction.

A 3 by 3 heatmap of causally masked attention weights with a zeroed upper triangle. Row the: 1.0 on itself, 0 elsewhere. Row cat: 0.33 on the, 0.67 on itself, 0 on sat. Row sat: 0.25, 0.25, 0.50. Rows still sum to 1 over the visible positions.

The lesson-3 toy example, recomputed with a causal mask. ‘the’ can see only itself, so its whole budget lands there (1.0). ‘cat’ splits its budget over ‘the’ and itself (0.33 / 0.67). ‘sat’, last in the sequence, is unchanged — it could already see the full prefix. Masked cells are exactly 0, not merely small. Toy example; visible weights rounded to two decimals. (calculated — source: Worked example in this module; masking as in Vaswani et al. (2017))

Why −∞ before the softmax, rather than cleaning up afterwards? Because both alternatives fail quietly. Zero out the scores and you have done nothing: e^0 = 1, so future tokens still collect real weight. Zero out the weights after softmax and each row no longer sums to 1 — the attention budget leaks, and the output values shrink for early positions. Adding −∞ to the scores is the one intervention that removes the future and lets softmax renormalize the budget over what remains.

Not every transformer masks. Bidirectional encoders — BERT is the classic — let every token attend both ways, because their job is understanding, not generation: they train by filling in blanked-out tokens, where context on both sides is legitimate evidence. Decoder LLMs give up that luxury to gain the ability to generate. The mask also hands inference a gift: a past token’s key and value never change once computed, so serving systems compute them once and store them — the KV cache, which the Inference domain treats in full. One quirk to file away: every row must spend its full budget of 1 somewhere, so when nothing in the prefix is relevant, models often dump weight on the first token — the Attention sink phenomenon.

In production

Attention is the quadratic term in a transformer’s cost: scores form between every pair of visible tokens, so processing a prompt does work proportional to its length squared inside every attention layer. That mechanism — not any one vendor’s pricing — is why prompt length is never free.

AWS

Whether you self-host on GPU instances or call a managed endpoint such as Amazon Bedrock, the prompt is paid for during prefill: time-to-first-token and accelerator-seconds per request grow faster than prompt length, which is the mechanism behind input tokens being metered separately from output tokens. Capacity plans that assume a flat cost per token will undersize for long-prompt workloads.

Azure

On Azure AI Foundry deployments the same arithmetic sets your effective throughput: reserved capacity is consumed by prompt tokens as well as generated ones, and long-context requests eat disproportionate compute inside the attention layers. When a latency SLO is at risk, trimming what you stuff into the context usually moves time-to-first-token more than changing the deployment.

Google Cloud

On Vertex AI or self-managed GPU/TPU serving, doubling the prompt roughly quadruples attention-score compute during prefill, and longer contexts hold more per-request state in accelerator memory — shrinking the batch a single chip can serve just as per-request compute rises. The evergreen rule on every cloud: send the model what it needs, not everything you have.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.