Training Models to Think Longer
Lesson 2 of 3 in RL for Reasoning Models.
Chain-of-thought started life as a prompting observation: ask a model to write out intermediate steps before answering, and accuracy on multi-step problems improves. For years that was a trick you applied from the outside — a phrase in the prompt, some worked examples. Reasoning training turns the trick into a policy.
The mechanism is the RL loop from the last lesson, left to run. Attempts that work through the problem carefully — deriving intermediate results, checking them, trying another route when one stalls — pass the checker more often than attempts that leap to an answer. The checker rewards the hits, the optimizer reinforces whatever produced them, and step by step the model’s default behavior shifts toward spending more tokens thinking before it commits. Nobody writes “think longer” into a loss function. Length grows because length, spent on actual reasoning, wins.
The DeepSeek-R1 report shows this directly: its training curves for R1-Zero show average response length growing steadily across RL steps, alongside rising accuracy on the report’s evaluation tasks — the model, in the report’s framing, learning to use more thinking time to solve harder problems. The report also describes qualitative behaviors emerging over training: the model re-evaluating its own intermediate steps, checking its work, backtracking to try alternative approaches — including a moment the authors literally label an “aha moment”, where a mid-training checkpoint interrupts its own derivation to flag a mistake and start over. Treat the specifics as what one documented run reports rather than a law of nature — but the direction of the finding is the headline: self-checking and backtracking were not hand-programmed; the report describes them emerging from reward pressure alone.
Bar chart with three bars of illustrative token counts. A standard assistant’s full response is 180 tokens. A reasoning model answering the same question produces 1,400 tokens of thinking followed by a 220-token final answer, showing that the hidden reasoning segment can be several times larger than the visible reply.
Here is the fact this whole module pivots on, and it is architectural, not behavioral: thinking tokens are ordinary output tokens. There is no separate “reasoning engine” bolted onto the transformer. The thinking segment is produced by the same next-token loop as the answer — one forward pass per token, each one appended to the Context window, each one growing the KV cache, each one metered as output. The model is not pausing to think; it is writing, and you are paying for the manuscript. The tags around a thinking segment are formatting the model was trained to emit — a convention, not a mechanism. Architecture in Production prices the per-token forward pass; this is that lesson, multiplied.
Prompted CoT
Where the reasoning comes from: your prompt. You ask for steps, provide worked examples, or add a “think it through first” instruction.
- Works on any capable Instruction-tuned model — no special training required
- Depth and length track what you asked for; forget the instruction and the behavior disappears
- Reasoning appears inline in the visible answer, in whatever format you requested
- Quality ceiling: the model imitates the style of reasoning; nothing ever trained it that careful reasoning is what earns reward
Trained long CoT
Where the reasoning comes from: the policy. RL against verifiable rewards made extended reasoning the model’s default behavior.
- The model decides how long to think, scaling effort with perceived difficulty — no prompt engineering required
- Behaviors like self-checking and backtracking appear unprompted (as the R1 report describes them emerging)
- Thinking is typically delimited in a separate segment or channel, distinct from the final answer
- The reasoning is load-bearing: it was reinforced because it changed answers, not because it looked good
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.