Penalties, Stops, and Constraints
Lesson 3 of 4 in Decoding and Sampling.
Reshaping and truncation act on the distribution as the model produced it. A second family of knobs edits the Logits based on what has already been generated — history-aware nudges aimed at the repetition failure from lesson one.
Three variants dominate, and the mechanism differences matter. A repetition penalty scales the logit of any token that has already appeared, multiplicatively pushing it down (or, for negative logits, further down) each step it stays in view. A presence penalty subtracts a flat amount from every token that has appeared at least once — one strike and the tax applies, no matter how often the token recurred; it nudges the model toward introducing new tokens. A frequency penalty subtracts an amount proportional to the count of prior appearances — the more a token has been used, the harder it gets pushed down; it targets the loop dynamics directly, where each repetition would otherwise raise the odds of the next. APIs typically expose one or two of these under provider-specific names and scales, which is another reason settings do not transfer between stacks unexamined.
All three share a blind spot: the penalty cannot tell legitimate repetition from degenerate repetition. Code that must reuse an identifier, a contract that must repeat a defined term, a JSON array of similar objects — every one of those needs tokens to recur, and an aggressive penalty fights exactly that, producing misspelled variable names, drifting synonyms for defined terms, or malformed structure. Penalties are seasoning, not structure.
| Knob | What it actually does | Reach for it when | Failure mode when abused |
|---|---|---|---|
Repetition penalty | Scales down logits of tokens already in the context, every step | A model prone to verbatim loops in long generations | Set high, it degrades required repetition — identifiers, names, terminology mutate mid-output |
Presence penalty | Flat subtraction from any token that has appeared at least once | Nudging topical variety — pushing the model to introduce new words | The model swerves around common function words and phrasing, producing stilted prose |
Frequency penalty | Subtraction that grows with each repeat of a token | Damping the self-reinforcing loop where repetition breeds repetition | Long structured outputs (tables, JSON) starve — the format’s repeated tokens get taxed hardest |
Stop sequences | Serving layer halts generation when a configured string appears, truncating it from the output | Bounding turns in a template — cutting at a delimiter, a closing tag, a role marker | A stop string that can occur inside legitimate content silently amputates valid output |
Max tokens | Hard cap on output length — generation ends when the budget is spent, mid-sentence if need be | Cost and latency guardrails; every production call should set one | Set too low, output truncates mid-structure — the classic half-finished JSON that fails to parse downstream |
Stop sequences and max tokens are not distribution knobs at all — they are the serving loop’s exit conditions, and they end more generations than any sampled end-of-sequence token does. A stop sequence is checked against the generated text each step; when it matches, the engine stops decoding and returns everything before the match. Max tokens is the unconditional backstop: it bounds worst-case cost and per-token latency exposure for a runaway generation, which is why leaving it unset in production is an unforced error. Remember that a truncated-by-budget response and a naturally finished one can look similar to a casual reader — check the finish reason your API returns, not the vibes of the text.
One more mechanism deserves a paragraph because it subsumes a whole category of prompt engineering: constrained decoding. Instead of asking the model for valid JSON, the serving layer can enforce it — compile a grammar or JSON schema into a state machine, and at every step mask the logits of all tokens that would violate it, so only structurally legal tokens can be sampled at all. The output is then syntactically valid by construction: the model cannot emit an unclosed brace because the unclosed-brace token never survives the mask. What constrained decoding cannot buy is semantic correctness — the schema-valid answer can still be wrong, and a model forced down a grammar path it finds unlikely can produce degraded content inside a perfect envelope. This logit-masking mechanism is what tool-calling and structured-output APIs are built on; the wire formats and orchestration around them are the sister AI Agent Academy’s territory.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.