The Silent Model Swap

A JSON pipeline rots for three weeks after a model version update nobody asked for — retries and live prompt surgery make it worse, and the fix is a pin, a gate, and a hard schema boundary.

A composite teaching case: realistic fiction assembled from well-documented public patterns — not a real engagement.

The pipeline was the boring kind that works: claims documents and customer emails in, structured JSON out. A model deployment on Azure AI Foundry did the extraction — a System prompt with format instructions and a few worked examples, then JSON.parse on whatever came back, then straight into the claims system. Roughly forty thousand documents a day. Parse failures ran around 0.3%, absorbed by a single retry, and nobody had looked at the service dashboard in months because there was nothing to see. We tell this story because of what we did not do at setup time. The deployment had been created in an afternoon, a quarter earlier, with every default accepted — including the one that decides when the model underneath it changes. Nobody in the room that afternoon would have described accepting a default as a decision. It was one.

The symptom arrived on no particular day. Parse-failure alerts, previously a monthly curiosity, started firing weekly, then daily. Over about three weeks the failure rate crept from 0.3% to 1%, then past 4%. The failures were maddeningly intermittent: a document would fail, retry, and pass — because with Temperature above zero, Sampling means the model does not produce the same output twice, and sometimes the second draw happened to be parseable. On-call wrote it up as "model flakiness" three shifts in a row. The deploy log was the alibi everyone kept reaching for. No release in the window. No config change, no prompt change, no dependency bump. Every incident review opened with the same sentence — nobody changed anything — and closed without a cause.

Two wrong turns cost us most of a month. Wrong turn one: retry harder. Since retried documents often passed, the obvious move was more retries — one attempt became three, with backoff. The dashboard improved immediately, which felt like progress and was actually anesthetic. Retries treat sampling noise; they cannot treat a shift in what the model systematically tends to produce. Every failed attempt re-bought the full prompt and completion, so the token bill climbed with the failure rate — we were paying extra, per document, to hide our own signal. The underlying trend kept worsening for another week before anyone noticed, because the metric everyone watched was failures after retry. Wrong turn two: prompt surgery in production. When retries stopped covering it, we started editing the live system prompt. "Return ONLY valid JSON." Then "No markdown." Then a regex hotfix downstream to strip code fences. Each tweak went straight to production, judged by eyeball on the next hour of traffic. Some tweaks helped one document class and hurt another; none were tested against anything; two of them conflicted. We were negotiating with a trained policy we could not see, one anecdote at a time — and mutating the only stable reference point we had left.

The diagnosis took an afternoon once someone changed the question. Instead of asking how many outputs failed, an engineer pulled a sample of failing outputs and diffed them against logged outputs from two months earlier for similar documents. The failures had signatures, and the signatures were consistent: markdown code fences wrapped around otherwise-valid JSON, a sentence of preamble ("Here is the extracted record:"), an occasional helpful note appended after the closing brace. The old outputs were bare JSON, byte one to byte last. That is not flakiness. That is a different habit. Habits have owners. A model's default output format — fences or no fences, preamble or silence — is not in the API contract and not in our prompt; it is a disposition learned during post-training, from SFT demonstrations and Preference data that happened to reward one presentation style over another. A new Model release is a different post-training run, and different post-training means different habits, even when the marketed capabilities only improve. The module on what post-training does and does not promise gives this its proper name: a model swap is a behavior-profile swap. We had built a parser on top of a behavior nobody had ever promised us. One console check confirmed the mechanism. Our deployment's model version had rolled forward — weeks earlier, right where the failure curve left its baseline. On Azure AI Foundry a deployment binds a model version, and an upgrade policy on the deployment decides whether new versions roll in automatically or wait for you. Ours was set to roll. We had never chosen that; we had accepted it, in that afternoon a quarter ago, without reading it.

The fix came in three layers, each catching what the previous one misses. Layer one: pin the version where the platform allows it. We set the deployment's upgrade policy so that new model versions wait for us instead of rolling in. Pinning is deferral, not exemption: providers retire model versions, so Deprecation turns every pin into a migration with a deadline. We put the published retirement dates on the team calendar the same day. A pin without a migration plan is just a slower incident. Layer two: a regression harness that gates every migration. We built a Golden set of about five hundred redacted production documents with reviewed expected extractions, and an Eval harness that runs any candidate — new model version, new model, or meaningful prompt change — against it before production sees it. The gate asserts three things: parse rate on first attempt, schema conformance, and field-level accuracy against the expected extractions. It reports diffs, not just scores, because the dangerous changes are behavioral: a candidate can match on accuracy while moving every date into a different format. This is Regression testing as the production evals module teaches it — the suite exists before the swap does, or it is archaeology instead of a gate. Layer three: schema validation as a hard boundary. The lasting error was architectural: JSON.parse straight into the claims system means model output was trusted input. It never should have been. Model output is untrusted input to whatever consumes it — that is the OWASP LLM05 improper output handling finding, taught in the Top 10 module, and it holds even when nothing malicious is happening; a version bump was enough. We now validate every response against an explicit JSON schema at the pipeline boundary. A validation failure gets exactly one structured repair attempt — re-prompting with the validator's error — and then goes to a dead-letter queue with the raw output attached for a human. No blind resample loops, ever again: the validator's first-attempt failure rate is the alert metric, so the next behavioral shift shows up in hours, not weeks.

What changed, in the end, was less the architecture than the bookkeeping. First-attempt parse failures went back under the old baseline once the prompt was re-tested as a whole against the pinned version — and the fence-stripping regex came out, deliberately, with a gate run to prove it safe to remove. Retry spend fell to near zero because retries went back to being what they are for: transient faults, not distribution shift. The gate has since carried two real migrations. One passed cleanly and re-pinned the same week. The other failed usefully: the candidate version renamed its habits again — field casing shifted in a way that would have silently broken a downstream mapping — and the fix happened in a branch, against the candidate, before rollout, instead of in production during an incident. Model version upgrades now appear in sprint planning next to library upgrades, with a deadline supplied by the provider's deprecation schedule and a cost supplied by the gate. The bug was never that the model changed. Hosted models change; that is the deal. The bug was that our system had no seam where the change could be noticed, tested, and either absorbed or refused. Now it has three.