Sampling and temperature: where the dice live

Lesson 3 of 5 in LLM Mechanics for Agent Builders — No Math Required.

A common mental model says the LLM “writes a reply.” Mechanically, it does something stranger: for each position, it produces a probability distribution over every token in its vocabulary — maybe the at 41%, a at 22%, this at 9%, and tens of thousands of tiny alternatives. Then a separate step, the sampler, rolls the dice and picks one. The chosen token is appended, and the whole process repeats for the next position until a stop token ends the reply.

Two knobs control that dice roll. Temperature reshapes the distribution: at 0 the sampler always takes the single most likely token (greedy decoding); higher values flatten the distribution so less-likely tokens win more often — more variety, more surprise, more nonsense at the extreme. Top-p trims the long tail before sampling, so the dice only land on tokens that make up the most plausible slice.

How one reply gets written — token by token

  1. Context window in

    System prompt, transcript, tool results — everything the model can see for this call.

  2. Model outputs a probability distribution

    A score for every token in the vocabulary — this part is (nearly) deterministic computation.

  3. Temperature / top-p reshape it

    Temperature flattens or sharpens; top-p cuts the tail. This is configuration, not intelligence.

  4. Sampler picks ONE token

    The only dice roll in the system lives here. At temperature 0 it just takes the top-scoring token.

  5. Token appended to the output

    The pick becomes part of the input for choosing the next token — early picks steer everything after.

  6. Stop token?
  7. Reply returned

Temperature 0

Greedy: always the top token. Output varies as little as the serving stack allows — the natural setting for tool-call arguments, extraction, classification, and anything parsed by code downstream. But read the warning below: this reduces variance; it does not buy you determinism, and it does not make answers more correct — a confidently wrong model is confidently wrong at every temperature.

Low–moderate (~0.2–0.7)

Mostly-likely with some spread. Sensible for conversational replies, summaries, and drafting, where slightly varied phrasing is fine or even desirable. Many vendor defaults live here (illustrative — defaults differ by API and change; check yours, and set it explicitly rather than inheriting it silently).

High (~1 and up)

The tail gets a real vote. Useful for brainstorming and creative variety — ten genuinely different taglines instead of one tagline ten times. Risky inside an agent loop: an unlikely token in a tool call is not “creative,” it is a malformed argument or an action nobody wanted.

Key terms: sampling, temperature, top-p, non-determinism

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.