Choosing Settings You Can Defend

Lesson 4 of 4 in Decoding and Sampling.

There is no universally correct temperature — but there are defensible settings: choices you can explain in terms of what the task rewards, and that you have actually tested against your own traffic. The reasoning runs on one axis. When a task has one right answer to recover — extraction, classification, structured transformation — randomness is pure downside: any draw away from the model’s best guess is a chance to be wrong, so you push toward greedy (temperature at or near 0) and add structural enforcement. When a task rewards variety — brainstorming, creative drafting, generating alternatives — determinism is the downside: the argmax answer is the same safe answer every time, so you open the distribution up and often sample several candidates. Most production work sits between, and the honest answer is a starting point plus an eval: pick the bundle that matches the task family, then let measured quality on your own data move the knobs.

Treat every number that follows as a common starting point, not gospel — models, engines, and tasks differ, and the previous lessons explained why the same nominal settings can behave differently across stacks.

Pick a defensible starting bundle

Interactive decision tree — outcomes:

Deterministic extraction bundle
Temperature 0 (or your engine’s minimum), top-p left at 1, constrained decoding or the API’s structured-output mode doing the real enforcement, tight max tokens, explicit stop sequences. Randomness buys you nothing here; structure enforcement buys you parseability. Remember: temperature 0 reduces variance, it does not guarantee bit-identical reruns.
Grounded prose bundle
Low temperature (commonly somewhere around 0.2–0.3 as a starting point) with default top-p. Enough determinism that phrasing stays sober and claims stay close to the model’s best estimate, enough sampling to avoid greedy’s repetition rut on longer passages. Validate against a hallucination-sensitive eval before trusting it.
Assistant chat bundle
Moderate settings — providers’ shipped defaults typically land here for a reason (often mid-range temperature with top-p near 0.9–1). Start from your provider’s documented default rather than folklore, then A/B against your own conversations. Chat is where untested knob-twiddling most often ships as vibes.
Ideation bundle
Higher temperature (around 0.9–1.0 as a starting point), top-p around 0.95, and several samples per prompt — diversity across candidates is the product, and a human or a scoring step picks the winner. Watch the cost meter: n candidates means n× the output tokens.
Creative writing bundle
Temperature in the 0.7–1.0 range with top-p around 0.9–0.95 is a common opening position; penalties light or off, since good prose repeats deliberately. Tune by reading outputs, not by belief — small temperature moves change texture more than most prompt edits.

Temperature 0 is not bit-determinism

Temperature 0 means intend the argmax — it does not mean the same request always returns the same bytes. Floating-point addition is not associative, so the logits themselves can differ across runs when the serving stack sums in a different order: kernel selection, batch composition, hardware generation, and parallelism layout all vary run to run on a shared service, and in Mixture of experts (MoE) models routing adds its own sensitivity. When two candidate tokens sit close together, a last-decimal wobble flips the argmax — and one flipped token changes all context after it. Treat temperature 0 as low variance, not reproducibility. If you need stronger repeatability, pin what you can (model version, settings, seeds where the API offers them), and design downstream systems to tolerate variation rather than assume it away.

The last habit separates teams that can debug from teams that guess: document your settings like the code they are. A model output is a function of the model version, the prompt, and the full decoding configuration — temperature, truncation, penalties, stops, max tokens, and any structured-output mode. If any of those lives as an unversioned default someone set in a console, your system has an invisible input. Put the whole bundle in versioned configuration, log it alongside every response you store (finish reason included), and when behavior shifts, you can answer the first diagnostic question — what changed? — with evidence instead of archaeology. The same logging is what makes an eval reproducible: a quality score attached to unrecorded settings measures nothing.

In production

Sampling settings are deployed configuration: every managed platform accepts them per request, which means every request is an opportunity for drift unless the bundle is pinned, versioned, and reviewed like code.

AWS

On Amazon Bedrock, inference parameters ride along with each request, and different hosted model families name and bound their knobs differently — a bundle tuned for one model id is not portable to another. Keep the per-model parameter set in versioned configuration (Bedrock’s prompt-management tooling can hold parameters with the prompt), and treat any change — including a model-version move with “unchanged” knobs — as a deployment that gets an eval run before rollout.

Azure

On Azure OpenAI in Azure AI Foundry, apps call a deployment, and sampling parameters arrive per request from your code. That split is the trap: platform-side model-version updates on the deployment change behavior under settings you never touched. Keep the decoding bundle in application configuration or IaC next to the deployment name it was tuned for, and re-run your evals whenever either side — parameters or deployment — moves.

Google Cloud

On Vertex AI, the generation config (temperature, top-p and friends, max output tokens, stop sequences) is an explicit request object — which makes the discipline easy to automate: construct it from versioned config, never inline literals; log it with every stored response; and diff it in code review like any other behavioral change. Model-version rotation gets the same treatment: same knobs, new model is a new system until your evals say otherwise.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.