Causal Language Modeling

Lesson 1 of 3 in Objectives and the Loss Curve.

The previous module ended with a corpus: trillions of tokens of filtered, deduplicated text. This module is about what the model does with them — and the answer is the objective you already know from Foundations, Next-token prediction, rebuilt for throughput.

At inference time you picture the game one guess at a time: given a prefix, predict the next token. Training cannot afford to work that way. Here is the trick that makes it industrial. Take a training sequence of, say, 2,048 tokens. Do not think of it as one example — think of it as 2,048 examples stacked on top of each other. Position 1 must predict token 2 from token 1. Position 2 must predict token 3 from tokens 1–2. Position 500 must predict token 501 from the five hundred tokens before it. The target sequence is simply the input sequence shifted one position — the text is its own answer key. Nobody labeled anything; the supervision comes free with the data, which is why the field calls this self-supervised learning.

Two properties make it cheap enough to run at scale. First, Causal masking — the same mask you met inside the Transformer — guarantees position t cannot see anything to its right, so every position’s prediction is computed in one parallel forward pass: one pass, 2,048 simultaneous prediction problems. Second, Teacher forcing: at every position the model predicts from the true prefix of the document, never from its own earlier guesses. Errors cannot compound during training; each position is a clean, independent question about real text.

One toy sentence as six supervised examples. The target column is just the input shifted one position left — every prefix is a training question, and the document itself supplies the answer. Word-level tokens for readability; real models split subwords.
PositionContext the model seesTarget it must predict

1

The

cat

2

The cat

sat

3

The cat sat

on

4

The cat sat on

the

5

The cat sat on the

mat

6

The cat sat on the mat

⟨end-of-text⟩

Now multiply. A 2,000-token article is 2,000 predictions, each graded against a known answer. A trillion-token corpus is a trillion graded exercises. GPT-3 trained on roughly 300 billion tokens (Brown et al. 2020) — which is to say, roughly 300 billion supervised examples, with no annotator anywhere in the pipeline. This is the economic core of the LLM era: the only labeling scheme that scales to internet-sized data is the one where the data labels itself.

It also explains why the previous module obsessed over data quality. Every token in the corpus is a gradient. A duplicated document is the same exam question graded twice; a page of boilerplate is a thousand low-value exercises the model still pays full compute to practice. The objective is indiscriminate — it learns whatever the Data mixture feeds it, one shifted position at a time.

One training step, end to end

  1. Batch of token sequences

    Documents from the training mixture, tokenized and packed into fixed-length sequences.

  2. One forward pass — all positions at once

    Causal masking makes every position a legal prediction problem simultaneously: position t sees only tokens 1…t.

  3. Targets = the same sequence, shifted by one

    Teacher forcing: the target at position t is the true token t+1 from the document, regardless of what the model would have generated.

  4. Per-position cross-entropy

    At each position, −log of the probability the model assigned to the true next token — the same surprise measure from Foundations.

  5. Average into one scalar loss

    Mean over all positions and all sequences in the batch. This single number is what the loss curve plots.

  6. Backward pass + optimizer update

    Gradients flow through the whole network; the learning-rate schedule scales the step taken on every weight.

  7. Repeat — for weeks, over trillions of tokens

    Pre-training is this loop and nothing else. Everything later in this domain — parallelism, precision, budgets — is engineering to keep this loop fed.

Key terms: Teacher forcing, Causal masking, Cross-entropy, Masked language modeling (MLM), Loss spike

The loss over positions, written out

For one training sequence x₁ … x_T, the objective is the average Cross-entropy over its positions:

L = −(1/(T−1)) · Σₜ log p_θ(x_{t+1} | x₁ … x_t) for t = 1 … T−1

Each term is the surprise −log p(true token) you met in Next-Token Prediction, Really — the same quantity, reused at every position and averaged. Average again over the batch and you have the number every training chart plots: per-token Loss, conventionally in nats. Its exponential, e^L, is the training Perplexity.

Three details are worth having straight. The average is per token, not per document — a 10,000-token paper and a 100-token comment contribute in proportion to their length, and every token counts equally, boilerplate and brilliance alike. One forward pass produces every term: the network emits Logits at all T positions at once, the Softmax and −log are evaluated at each, and a single backward pass propagates the summed gradient — this is why training compute is counted per token processed, the accounting the Compute Budgets module builds into C ≈ 6·N·D. The floor is not zero: language is genuinely uncertain (‘My favorite color is ___’), so even a perfect model pays the irreducible entropy of text. Lesson three shows where published work puts that floor.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.