Triage drill: which rung, which fix

Lesson 5 of 5 in Structured Outputs: Getting Data, Not Prose.

Ten failures land in your logs. Route each one. The judgement you are practising is the whole module compressed into one question: is this noise, a shape problem, a truth problem, a stop-everything problem — or a schema I should never have shipped?

The fifth bucket is the one experienced teams reach for most. A failure that recurs on 20% of inputs is not a retry candidate; it is a design defect the retry path is politely hiding.

Interactive sorting exercise: Sort each failure to the cheapest response that can actually fix it. When a failure is systematic rather than incidental, retrying is the wrong tool entirely.

Every card in that stack should end up in a file. A structured-output defect is unusually cheap to capture — the raw model output, the schema version, and the expected verdict are all you need — and unusually valuable to keep, because the two changes most likely to break your parsing are the two changes you will make most often: a prompt edit and a model swap.

Wire those fixtures into CI and the question “did the new model still respect our schema?” stops being a vibe and becomes a number. That is the handoff to the evals domain: a parse-failure corpus is the easiest golden dataset you will ever build, and the only one that pays for itself the first week.

Interactive flashcard deck.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.