The ladder of enforcement
Lesson 1 of 5 in Structured Outputs: Getting Data, Not Prose.
Inside an agent loop, a model’s output is rarely for a human to read. It is an argument list for a tool call, a routing decision, a row for a database, a field on a ticket. The moment the output feeds code instead of eyes, prose is a liability: your program needs {"priority": "p1"}, and the model is perfectly capable of returning “I’d classify this as a P1 — it looks urgent!”
So you ask for JSON. The question this lesson answers is how hard the ask is. There are three rungs, they differ by who enforces the shape, and the difference between them is the difference between a bug you catch in staging and a 3 a.m. page.
Key terms: structured outputs, JSON Schema, JSON mode, constrained decoding, function calling, schema validation
The enforcement ladder
- Schema-constrained decoding — the sampler cannot emit a non-conforming token
Your JSON Schema is compiled into a grammar, and at every decoding step the runtime masks out tokens that could not continue a conforming document. Non-conforming output is not corrected after the fact — it is unrepresentable. Shipped under names like structured outputs, strict schemas, or grammar-constrained decoding, both in provider APIs and in open-source samplers.
The strongest rung, with two honest caveats: every implementation supports a subset of JSON Schema (exotic
oneOf, unbounded regex and recursive refs are the usual casualties), and a run that hits its token limit still ends mid-document — truncated JSON parses no better for having been constrained. - JSON mode — syntax guaranteed, shape not
A provider flag that constrains generation to syntactically valid JSON. No prose preamble, no ```json fence, no trailing apology. What it does not promise is your keys, your types, or your enum values —
{"answer": "p1 maybe"}is impeccable JSON and useless to your parser.Cheap, near-universal as of September 2026 (names and flags vary — check the docs), and a real improvement over rung one because it eliminates an entire class of noise. Treat it as syntax insurance, never as a contract.
- Prompt and pray — the model is asked nicely
“Respond with only a JSON object matching this shape.” Compliance is high on easy inputs and degrades exactly when you need it: long contexts, ambiguous inputs, retries, a model swap. The failures are boring and endless — fenced code blocks, a chatty preamble, a trailing note, single quotes, a trailing comma, an extra field, an invented enum value.
Still the only rung available for some models and some endpoints, so your parser must handle it. Never the only layer.
Climbing the ladder buys you shape, and it buys it cheaply — a strict schema flag costs one line of config and deletes most of your parsing folklore. That is why the correct default is: climb as high as your provider and your schema allow, then keep every layer below. The rungs are cumulative, not alternatives.
What climbing does not buy you is a smaller validator. The guarantee stops precisely at the grammar’s edge, and your business rules live outside it.
| Rung | What is guaranteed | What still breaks | Flexibility | Availability (as of Sep 2026 — verify) |
|---|---|---|---|---|
Prompt and pray | Nothing. Compliance is a probability that drops with context length and input weirdness. | Fences, preambles, trailing commentary, invalid JSON, wrong keys, invented enum values. | Total — any shape you can describe in words, including shapes no schema language expresses. | Everywhere, including models with no structured-output features at all. |
JSON mode | The output parses as JSON — if generation completes. | Keys, types, enums, required fields, nesting: all still up to the model. | High — you are constrained to “some JSON”, not to a particular document. | Widely available on hosted chat APIs under provider-specific flag names; confirm in current docs. |
Schema-constrained decoding | The output conforms to the compiled schema by construction — keys, types, enums, required fields. | Semantics: right shape, wrong values. Plus truncation at token limits, refusals, and unsupported schema features. | Lower — you must express the contract in the supported JSON Schema subset, so some designs need reshaping. | Common on frontier hosted models and in local grammar samplers; the supported subset differs per implementation and moves. |
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.