Key takeaway: Schema conformance can be enforced during generation rather than checked afterwards. When the mechanism is available, retry-on-parse-failure is the wrong architecture.
The Failure Rate Nobody Budgets For
A prompt requesting JSON produces valid JSON in most cases. The remainder arrives with a markdown fence around it, a trailing comment, a truncated closing brace, or a helpful sentence before the object.
At one percent failure and a hundred thousand daily requests, that is a thousand failures per day. Handling them with retries multiplies cost and latency for exactly the requests that were already problematic.
How Constrained Decoding Works
At each generation step the model produces a distribution over the vocabulary. Constrained decoding masks tokens that cannot appear next given the schema and the text produced so far.
After {"status": the grammar permits only tokens that begin a valid value for that field. If the schema declares an enum of three strings, only tokens starting one of those three are available. The model cannot emit invalid output because invalid tokens have zero probability.
This is a hard guarantee rather than a strong tendency. The output parses because it could not have failed to parse.
| Approach | Conformance | Cost |
|---|---|---|
| Prompt asking for JSON | ~95–99% | Baseline |
| Prompt + retry on failure | ~99.9% | Extra calls on failures |
| Provider structured output mode | 100% for shape | Baseline |
| Local constrained decoding | 100% for shape | Slight overhead |
What It Does Not Guarantee
Structure is not correctness, and conflating them causes real bugs.
A schema guarantees total is a number. It does not guarantee the number is right. It guarantees category is one of your enum values, not that it is the applicable one. It guarantees a date field is a string matching a pattern, not that the date exists — 2026-02-31 satisfies most date patterns.
Semantic validation remains entirely your responsibility: cross-field consistency, referential integrity, business rules, plausibility ranges. Constrained decoding removes parsing failures from your error budget so you can spend attention on these.
Designing Schemas the Model Handles Well
Flat structures outperform deeply nested ones. Extracting six top-level fields is more reliable than one object nested four levels deep, and it is easier to validate.
Prefer enums over free strings wherever the valid set is known. This is the highest-value schema decision available, because it eliminates an entire class of downstream normalisation.
Include field descriptions in the schema. Providers pass them to the model, and a description saying “total including tax, in minor units” prevents a recurring category of misinterpretation.
Add an explicit null or unknown option for fields that may be absent. Without one, the model must invent something to satisfy the schema, which converts a missing value into a fabricated one — a strictly worse outcome.
The Bottom Line
Use provider structured output modes or local grammar constraints so parse failures stop existing. Keep schemas flat, use enums aggressively, describe every field, and permit explicit unknowns. Then invest the recovered effort in semantic validation, which no schema can perform.
