Chain of Thought — and Its Limits

Lesson 3 of 4 in Prompting as Engineering.

Some tasks fail not because the model lacks the knowledge, but because the answer requires several dependent steps and the prompt asks for the conclusion in one leap. Chain-of-thought (CoT) prompting attacks exactly this: instead of demonstrating question → answer, you demonstrate question → worked steps → answer, and the model imitates the shape — generating intermediate reasoning before its conclusion.

Wei et al. (2022) introduced the technique and reported that it substantially improved performance on arithmetic, commonsense, and symbolic multi-step benchmarks — with a crucial qualifier: the gains appeared for sufficiently large models, and smaller models often produced fluent-looking steps without the accuracy improvement. As always with published results, the headline transfers as a mechanism, not a guarantee: whether CoT helps your task on your model is an empirical question for your eval set.

Why would asking for steps help? A useful (and honest) intuition: each generated step becomes part of the context for the next Token, so the model conditions on its own intermediate results instead of holding the whole derivation implicitly in one prediction. The generation itself acts as working memory.

Direct answer

Q: A warehouse ships 4 pallets. Each pallet holds 50 boxes,
each box holds 12 units, and 5% of units arrive damaged.
How many sellable units arrive?
A:

A typical failure — this transcript is invented for teaching:

The model multiplied correctly and skipped the damage step. One-leap prompts invite one-leap answers, and a dropped step is invisible in the output.

Chain-of-thought

Q: A warehouse ships 4 pallets. Each pallet holds 50 boxes,
each box holds 12 units, and 5% of units arrive damaged.
How many sellable units arrive?
A: Let's work through this step by step.

An invented, illustrative completion:

4 pallets × 50 boxes = 200 boxes.
200 boxes × 12 units = 2,400 units.
5% of 2,400 = 120 damaged units.
2,400 − 120 = 2,280 sellable units.
Answer: 2,280

Every intermediate result is now in the context, conditioning the next step — and a wrong step is at least visible, which the direct answer never offers.

Bar chart comparing illustrative output lengths: a direct answer of about 5 tokens versus a chain-of-thought answer of about 60 tokens for the same question, showing that chain-of-thought inflates output size roughly an order of magnitude in this toy example.

The bill for thinking out loud: illustrative output-token counts for the two answers above. The numbers are toy values for this invented example — but the shape is the point: CoT multiplies output tokens, and output tokens are the slow, serially generated, typically pricier kind. Estimate your own workloads with the token cost estimator at /tools/token-cost-estimator. (illustrative — source: Wei et al. (2022) — Chain-of-Thought Prompting Elicits Reasoning in Large Language Models)

CoT has two limits every engineer should carry around.

Verbosity is a cost, not a side effect. The steps are output tokens: they are generated one at a time during Decode, they dominate latency, and on most pricing sheets output tokens cost more than input tokens. Applying CoT to tasks that never needed multi-step reasoning — classification, lookup, simple extraction — buys you slower, pricier responses and no accuracy. Reserve it for tasks where dependent steps are genuinely the bottleneck, and let your eval set arbitrate.

The transcript is not the computation. The steps a model writes are generated text, produced by the same next-token machinery as everything else. Research on faithfulness has shown that stated reasoning does not necessarily reflect whatever internal computation produced the answer — a model can write plausible steps and reach its conclusion by another route, or write a wrong step and land on the right answer anyway. The honest position: CoT text is useful evidence for debugging and often improves answers, but it is not an execution trace, and building audit or compliance stories on it as if it were one is a category error.

The technique also stopped being only a prompt trick. Reasoning models are post-trained — typically with reinforcement learning on verifiably correct answers — to produce long internal reasoning by default, no “step by step” incantation required. How that training works is the RL for Reasoning Models module’s story; the adaptation-level consequence is that on such models, classic CoT prompting is partly redundant, and your lever shifts to how much thinking you request and are willing to pay for.

The faithfulness caveat, one level deeper

The claim “CoT may be unfaithful” has a precise shape. Faithfulness asks: is the stated reasoning causally responsible for the answer? Several published probes suggest the answer is often not fully. Perturbation studies edit or truncate a model’s stated steps and observe whether the final answer changes — sometimes it does not, implying the answer was not strictly downstream of those steps. Bias studies show models can be nudged toward an answer by irrelevant features of the prompt while their written rationale never mentions the nudge. Neither result says CoT is useless; both say the text is a generated account, optimized to be plausible next tokens, not a log of internal state.

Practical guidance follows directly. Use CoT to improve answers and to make failures inspectable — a visibly wrong arithmetic step is a gift when debugging. Do not use the absence of a visible error as evidence of a sound process, and do not present CoT transcripts as explanations with regulatory or audit weight. Where the magnitude of unfaithfulness matters to your application, measure it on your task; published findings establish the phenomenon, not your numbers.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.