The Sampling Knobs
Lesson 2 of 4 in Decoding and Sampling.
If maximizing is wrong for open-ended text, the alternative is honest randomness: sample the next token in proportion to its probability. Pure sampling has the opposite problem, though — the distribution’s long tail contains thousands of individually unlikely tokens whose combined mass is substantial, so an unlucky draw occasionally picks something incoherent, and one bad token becomes context for every step after it. Every knob in this lesson is a way to reshape or truncate the distribution before the draw, taming the tail without falling back into the argmax rut.
Temperature reshapes. Before softmax, every logit is divided by a value T. With T below 1, gaps between logits widen, probability concentrates on the leaders, and output becomes more predictable; as T approaches 0 the distribution collapses onto the argmax — which is why “temperature 0” is, in intent, greedy decoding. With T above 1, gaps shrink, the distribution flattens, and the tail gains mass. Temperature never reorders candidates — the most likely token stays most likely — it only changes how much the draw favors it.
Top-k truncates by count: keep only the k most probable tokens, renormalize, and sample among them (Fan et al., 2018, introduced it for story generation). It kills the tail by construction. Its weakness is that one fixed k must serve every step: when the distribution is peaked, k = 50 admits dozens of junk candidates; when it is genuinely flat — ten equally good continuations — a small k amputates legitimate choices.
Top-p, or nucleus sampling, truncates by mass: keep the smallest set of top tokens whose cumulative probability reaches p, renormalize, and sample from that “nucleus” (Holtzman et al., 2020). The cutoff now adapts — a confident distribution yields a nucleus of one or two tokens, an open one yields dozens — which is exactly the property top-k lacks, and why top-p became the workhorse truncation knob. Min-p is a later community refinement in the same spirit: keep tokens whose probability is at least some fraction of the top token’s, so the bar scales with the model’s confidence at that step.
Horizontal bar chart of an illustrative next-token distribution after the context “The weather today is”: “ sunny” 40%, “ cloudy” 25%, “ cold” 15%, “ mild” 8%, “ awful” 5%, and a final bar labeled “everything else — thousands of tail tokens” at 7%. The bars illustrate that top-k keeps a fixed count of tokens while top-p keeps however many are needed to reach a cumulative probability mass.
The temperature math, in one paragraph
The softmax with temperature is p_i = exp(z_i / T) / Σ_j exp(z_j / T), where z_i are the Logits. Dividing by T rescales every logit gap: if two tokens differ by Δ in logit space, their probability ratio is exp(Δ / T). At T = 1 you get the model’s learned distribution unchanged. Halving T squares every such ratio — a token that was 4× more likely becomes 16× more likely — which is why even modest reductions feel dramatically more deterministic. As T → 0 the largest logit’s share goes to 1 (greedy in the limit); as T → ∞ every ratio goes to 1 and the distribution approaches uniform over the vocabulary. Two properties worth internalizing: temperature is rank-preserving (it never changes which token is most likely, only by how much), and it acts before truncation in most pipelines — so a high temperature can inflate tail tokens enough to change what top-p’s cumulative cutoff keeps, one of the ways knob order changes results.
Key terms: Sampling, Temperature, Top-k sampling, Top-p (nucleus) sampling, Greedy decoding, Beam search
Tool: Sampling Lab — Feel these knobs instead of memorizing them: the Sampling Lab lets you drag temperature, top-k, and top-p against a live toy distribution and watch the nucleus grow, shrink, and shift.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.