RNNs and the Neural Turn

Lesson 2 of 4 in From N-grams to Transformers.

In 2003, Yoshua Bengio and colleagues published A Neural Probabilistic Language Model — the paper that moved language modeling from counting to learning. Same job as before, predict the next word; radically different machinery, built from two moves.

Move one: represent words as vectors. Instead of treating each word as an unrelated symbol, give every Vocabulary entry a short list of learned numbers — an Embedding. Nobody assigns these by hand: they are parameters, adjusted during training, and words that are used in similar ways drift toward similar vectors because that is what makes predictions better.

Move two: share the machinery. One neural network, with one set of Weights, computes next-word probabilities from the context’s vectors — the same weights for every context that ever occurs. Put the two moves together and something an n-gram could never do falls out: evidence transfers. The counting model that saw the cat sat a million times knew nothing about the dog sat; the neural model does, because cat and dog sit near each other in vector space, so contexts containing them produce similar predictions. One training sentence now improves the model’s behavior on a whole neighborhood of sentences it has never seen. Sparsity stops being fatal — and Bengio’s model beat the best smoothed n-grams of its day on Perplexity to prove it.

Bengio’s network still read a fixed window of previous words — a neural n-gram, in effect. The recurrent neural network (RNN) removed the window. An RNN reads left to right, one Token at a time, carrying a hidden state: a fixed-size vector that serves as its running memory of everything read so far. Each step folds the new token into the state; in principle, no context limit at all.

In practice, early RNNs forgot fast — the training signal connecting a word to consequences many steps later decays to nearly nothing (the deep dive below shows why). The long short-term memory network (LSTM — Hochreiter & Schmidhuber, 1997) added learned gates that control what enters, stays in, and leaves the memory, and made recurrent training work at useful depths. By 2010, recurrent models were beating n-grams at language modeling itself (Mikolov et al., 2010), and LSTMs went on to power speech recognition and machine translation through the mid-2010s.

An RNN, unrolled over “the cat sat …”

  1. Read “the”

    Each incoming word arrives as its embedding vector.

  2. Update state h₁

    The hidden state is one fixed-size vector — the network’s entire memory of the text so far.

  3. Read “cat”

    The same weights combine the previous state with each new token — shared machinery, every step.

  4. Update state h₂

    h₂ must be computed from h₁ — no way to start this step early. That dependency is the serial bottleneck.

  5. Read “sat”

    Information from “the”, two steps back, now survives only as whatever trace h₂ kept of it.

  6. Update state h₃

    Every step, the whole past competes for space in the same fixed-size vector.

  7. Predict the next token from h₃

    The state feeds a layer that scores every vocabulary entry — a probability for each possible next token.

The unrolled picture shows both of the RNN’s structural problems at once — and both matter for everything that follows.

The serial bottleneck. State 200 needs state 199, which needs 198. Nothing about a long sequence can be computed in parallel across its positions; training is a step-by-step crawl. This is exactly the workload that parallel hardware — the GPUs that were about to transform machine learning — is worst at accelerating.

Long-range forgetting. Everything the model knows about the text so far must fit in one fixed-size vector. A name mentioned five hundred words ago competes for the same few thousand numbers with everything that came after it. Gates help; they do not repeal the constraint. Hold both problems in mind — the next lesson is the story of one mechanism dissolving the second, then the first.

Why gradients vanish — and what gates do about it

Training an RNN means backpropagation through time: unroll the network across the sequence and push the error signal backward, step by step, to find how each weight should change — the same Loss-driven weight adjustment as any neural network, just applied along the chain. Each backward step multiplies the signal by a factor derived from the recurrent weights. Multiply a hundred factors whose typical size sits below one and the product collapses toward zero: the gradient vanishes, and words far in the past receive essentially no training signal even when they matter most. Let the factors sit above one instead and the product explodes.

The LSTM’s answer is a memory cell updated mostly additively, with learned gates deciding what to write, what to keep, and what to expose at each step. Addition, unlike repeated multiplication, gives the error signal a path along which it survives many steps — that is the entire trick, and it was enough to make recurrence practical for a decade. But “survives many steps” is not “survives arbitrarily many”, and nothing about gating touches the serial crawl: gated or not, step t still waits for step t − 1.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.