When to Pay for Thinking

Lesson 2 of 3 in Reasoning Models in the Landscape.

Here is the fact that makes reasoning an economics topic and not just a capability topic: thinking tokens are output tokens. The chain-of-thought a reasoning mode produces is ordinary generated text — whether the platform shows it to you, summarizes it, or hides it entirely. That has two mechanical consequences. On platforms that meter output tokens, thinking counts on the meter, so cost scales with how much the model thinks. And because a transformer decodes one token at a time, every thinking token is a sequential step in front of your answer, so latency scales with it too. The serving-side mechanics — why output tokens dominate latency and how batching interacts with long generations — are covered in serving in production.

How big is the multiplier? That is exactly the kind of number this site refuses to hardcode: it depends on the model, the task, and the effort setting, and it changes with every release. Treat the expansion factor as a measurement you own, not a constant you look up.

So when does thought earn its bill? A useful decision frame, stated with the hedge it deserves — these are tendencies to verify against your own Eval harness, not laws:

Tasks that tend to earn thinking tokens are multi-step and checkable. Debugging code against failing tests, math and quantitative derivation, multi-document analysis where conclusions must survive scrutiny, planning with constraints. The common thread: more intermediate work can genuinely change the answer, and you can verify that it did.

Tasks that rarely earn them are single-step transforms. Extraction of known fields, reformatting, classification, template filling. If a standard tier already hits your quality target, extra thinking buys latency and cost, not accuracy — and without a check, you cannot even tell whether the longer answer is a better one.

Where the lesson-one matrix showed per-request controls, route per request. A dial or a mode toggle means one deployment can serve both kinds of traffic: minimal effort for the extraction path, higher effort for the analysis path, escalation on failure. Where the control is a separate variant, the same routing happens at model-selection time instead. Either way, a single blanket setting for mixed traffic is the failure mode to design against.

Reasoning tier or standard tier?

Interactive decision tree — outcomes:

  • Standard tier

    The job is already done at target quality. Thinking tokens here buy latency and cost, not accuracy. Re-check when the task or the target changes.

  • Build the check before buying the thought

    Reasoning pays off where you can verify that it helped. Without a check you cannot distinguish a better answer from a longer one. Write the eval first — then rerun this tree.

  • Reasoning tier

    Multi-step, checkable, and latency-tolerant — the profile that tends to earn its thinking tokens. Measure the output-token expansion on real traffic and confirm the quality delta on your golden set.

  • One model, routed per request

    Use lower effort for the interactive path and escalate — higher effort, retry with more thought — on failure or on flagged-hard inputs. This is exactly what per-request dials exist for.

  • Split the traffic

    Serve the interactive path on a standard tier and reserve the reasoning model or variant for the async path. Routing still happens — just at model-selection time instead of per request.

Budgeting a reasoning workload, symbolically

Every number in a reasoning budget is volatile except the structure of the calculation, so build the budget as algebra and plug in measured values that you refresh. Per request, define:

  • I — input tokens (prompt, context, retrieved material)
  • O — answer tokens the caller actually consumes
  • k — the thinking expansion factor: thinking tokens divided by answer tokens, measured on your traffic at your effort setting
  • p_in, p_out — your platform’s current per-token rates for input and output

Then the per-request cost is roughly I·p_in + (1 + k)·O·p_out, and decode latency scales with (1 + k)·O because thinking tokens are generated sequentially before and around the answer. Three things follow from the shape of the formula alone. First, k multiplies the output term — the term that is usually the expensive and slow one — so reasoning cost is dominated by how much the model thinks, not by how much you send it. Second, k is a function of task, model, and effort setting, which is why per-request routing (varying the effort, and therefore k, by task) is the highest-leverage cost control this tier offers. Third, any platform fact — rates, meters, what counts as output — belongs in variables you re-verify, never in the formula.

Measure k from your serving logs (output-token counts with the mode on versus off, per endpoint), then explore the arithmetic with the Token Cost Estimator; the Cloud Model Selection Matrix helps when the routing decision spans platforms.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.