Masked, Span, and the Road Not Taken
Lesson 2 of 3 in Objectives and the Loss Curve.
Causal language modeling was not the obvious winner. In 2018 the most influential model in NLP was BERT, trained with Masked language modeling (MLM): take a sentence, hide about 15% of its tokens, and train the model to reconstruct the hidden ones from the context on both sides (Devlin et al. 2018). Bidirectionality is the whole point. To fill in ‘The cat ⟨MASK⟩ on the mat’, the model reads left and right — richer evidence per prediction than any left-only model gets, which is why masked models dominated understanding tasks like classification and question answering for years.
T5 then generalized the blank into a span (Raffel et al. 2019): span corruption drops out contiguous chunks of text, replaces each with a sentinel marker, and trains a Seq2seq encoder–decoder to emit the missing spans in order. The same paper built its training corpus, C4, by aggressively cleaning Common Crawl — the pipeline story you already know. Three objectives, three bets on what a language model should practice.
Causal LM
Sees: The cat sat on the — left context only.
Predicts: mat — and the same rule applies at every position of every document, so every token yields a loss term.
Generation: run the training rule forward, one token at a time. The objective is the generation loop.
Masked LM (BERT)
Sees: The cat ⟨MASK⟩ on the mat — the full sentence, both directions, with ~15% of tokens hidden.
Predicts: sat — only the masked positions produce a loss.
Generation: no natural loop. The model fills blanks in text that must already exist; there is no left-to-right rule to run forward.
Span corruption (T5)
Sees (encoder): The cat ⟨X⟩ the mat — a corrupted sentence with spans replaced by sentinels.
Predicts (decoder): ⟨X⟩ sat on ⟨Y⟩ — the missing spans, emitted in order.
Generation: works within the fill-in format the model was trained on; open-ended continuation is not the native task.
So why did causal win for LLMs? Four arguments, in rough order of weight.
The objective is the product. A causal model’s training task — continue this text — is literally the deployment interface. Chat, code completion, drafting: all are continuations. Masked models predict blanks in text that already exists; to generate with one you must contrive an iterative unmasking scheme that the model was never really trained for. Causal models factor the probability of a sequence left-to-right, so Sampling from them is just running the training rule forward.
Every token is a training signal. A causal pass collects loss from all positions; a masked pass collects it only from the masked ~15%. For the same tokens through the machine, the causal objective extracts several times more supervised signal — a commonly cited efficiency argument for the decoder-only recipe at scale.
Simplicity scales. One stack, one mask, one loss, no encoder–decoder plumbing, no sentinel vocabulary. When the engineering challenge is keeping thousands of accelerators busy for months, the objective with the fewest moving parts wins ties.
Scale rewarded it. GPT-2 and then GPT-3 showed that pure next-token training, pushed far enough, yields models that follow instructions from examples in the prompt (Radford et al. 2019; Brown et al. 2020). Once in-context learning emerged from the causal recipe, the field consolidated around it.
Encoder models did not die — bidirectional encoders still power embedding, retrieval, and classification systems everywhere, and you will meet them again when the Adapting LLMs domain covers embeddings and RAG.
| Objective | What is predicted | Sees future context? | Generates naturally? | Canonical family |
|---|---|---|---|---|
Causal LM | The next token, at every position | No — causal masking hides everything to the right | Yes — generation is the training rule run forward | GPT lineage; nearly all modern generative LLMs (decoder-only) |
Masked LM | The original tokens at ~15% masked positions | Yes — attention runs over the whole sequence, both directions | Not naturally — it fills blanks; no left-to-right factorization to sample from | BERT and encoder-only kin |
Span corruption | Dropped-out spans, reconstructed in order by a decoder | Encoder sees the whole corrupted input | Within the fill-in format — trained to emit missing spans, not open-ended continuations | T5 (encoder–decoder) |
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.