Table of Contents
- The 97% Problem
- Why Prompting Alone Fails
- Constrained Decoding Changes the Guarantee
- Schema Design Affects Accuracy
- Valid Is Not the Same as Correct
- Handling Uncertainty in the Schema
- Retry Strategy That Converges
- Validation Layers
- Common Pitfalls
- Conclusion
- Frequently Asked Questions
Key takeaway: Prompting for JSON gives you high-probability compliance. Constrained decoding gives you a guarantee. The difference matters enormously at scale, because a 3% failure rate on a million requests is thirty thousand broken responses.
The 97% Problem
You ask a model for JSON. You specify the format precisely. You add “respond only with valid JSON, no markdown.” It works.
It works 97 percent of the time. The other three percent produce a response wrapped in a code fence, or prefixed with “Here is the JSON you requested,” or containing a trailing comma, or with an unescaped quote inside a string value, or truncated because the response hit a token limit mid-object.
Three percent sounds tolerable until you consider what it means operationally. At a thousand requests daily, that is thirty failures — enough to matter, few enough that they surface as mysterious intermittent bugs rather than an obvious problem. Each one is a parse exception somewhere downstream, and the stack trace points at your JSON parser rather than at the model.
The insidious part is that this failure rate is high enough to break things and low enough to pass testing. Five manual tests will all succeed. The problem appears in production, distributed across weeks, attributed to everything except the real cause.
Why Prompting Alone Fails
Understanding the mechanism clarifies why better instructions have limited returns.
A language model generates one token at a time, each sampled from a probability distribution over the vocabulary. Nothing in that process enforces syntactic structure. The model has learned that JSON-like contexts are usually followed by JSON-like tokens, which is a strong statistical tendency rather than a constraint.
Several situations push it off track:
Instruction competition. A long prompt with many requirements dilutes attention on the format instruction. The formatting requirement competes with everything else you asked for.
Content that resembles prose. When a string value contains something conversational, the model’s tendency toward natural language can reassert itself mid-generation.
Token limits. If generation is truncated, you receive a syntactically invalid prefix. This is not a model error at all, and it is one of the more common causes.
Training toward helpfulness. Models are trained to be conversational, which produces the “Here is your JSON:” preamble. That behaviour was deliberately instilled and it conflicts with machine-readable output.
Escaping edge cases. Nested quotes, newlines, and unicode inside string values require correct escaping. This is where subtle invalidity most often appears.
Better prompting reduces the failure rate — from perhaps 10 percent to 3 — and cannot eliminate it, because the mechanism permits any token at any position.
Constrained Decoding Changes the Guarantee
The reliable solution operates at the sampling layer rather than the instruction layer.
Constrained decoding restricts which tokens can be sampled at each step to those permitted by a grammar. If the schema requires an object, only { is permitted first. Inside an object, only a valid key string or } is permitted. After a key, only :. Tokens that would produce invalid output have their probability set to zero before sampling.
The result is a structural guarantee rather than a probability. Invalid JSON becomes impossible, not unlikely.
This is what underlies the structured output and JSON schema modes now offered by most providers, and it is why they behave qualitatively differently from prompting. When available, use them — the difference is not incremental.
# Prompting: high probability of valid output
response = model.generate(
"Extract the invoice fields as JSON. Output only JSON."
)
data = json.loads(response) # will raise, occasionally
# Constrained: structural guarantee
response = model.generate(
"Extract the invoice fields.",
response_format={"type": "json_schema", "schema": INVOICE_SCHEMA},
)
data = json.loads(response) # cannot raise on syntaxTwo caveats worth knowing. Constrained decoding guarantees syntax, not semantics — you will receive a valid object with the right shape, and the values may still be wrong. And heavily constrained generation can occasionally reduce output quality, because forcing the model into a rigid structure removes the freedom it would otherwise use to reason. The usual mitigation is permitting a free-text reasoning field before the structured fields.
Schema Design Affects Accuracy
An underappreciated point: how you write the schema changes how accurately the model fills it.
Use descriptive field names. invoice_total_including_tax produces better extraction than amt2. The field name is part of the prompt, effectively, and the model uses it to determine what belongs there.
Add descriptions to every field. Most schema formats support a description, and models attend to them. This is the cheapest accuracy improvement available.
Prefer enums over free strings for categories. An enumerated set of permitted values eliminates an entire class of normalisation problems — no more deciding whether “Pending”, “pending”, and “PENDING” are the same status.
Keep nesting shallow. Deeply nested structures produce more errors than flat ones. Two levels is comfortable; five is asking for trouble.
Order fields to support reasoning. Models generate sequentially, so a field that depends on another should come after it. Placing a conclusion field before the evidence fields forces the model to commit before reasoning.
{
"type": "object",
"properties": {
"evidence": {
"type": "string",
"description": "Quote the exact text supporting the classification"
},
"reasoning": {
"type": "string",
"description": "Brief explanation of why this category applies"
},
"category": {
"type": "string",
"enum": ["billing", "technical", "account", "other"],
"description": "The single best-fitting category"
},
"confidence": {
"type": "number",
"description": "0.0 to 1.0, how certain the classification is"
}
},
"required": ["evidence", "reasoning", "category", "confidence"]
}The ordering here is deliberate. Evidence and reasoning precede the category, so the model works through the problem before committing to an answer. Reversing that order measurably reduces accuracy on ambiguous cases.
Valid Is Not the Same as Correct
Constrained decoding solves syntax. The remaining failures are semantic, and they are more dangerous because they parse successfully.
The categories to expect:
Hallucinated values in required fields. A required field forces the model to produce something. If the source document lacks that information, it will invent a plausible value rather than fail. This is the single most consequential failure mode with strict schemas.
Type-valid nonsense. A date field containing a syntactically valid but impossible date. A number field with a value off by an order of magnitude.
Wrong-field assignment. Values correctly extracted and placed in the wrong fields, particularly when several fields have similar types.
Silent truncation. A list that should contain twelve items containing eight, with no indication anything was omitted.
The mitigation for the first and most important of these is making optionality explicit. Fields that may legitimately be absent should be nullable, and the description should state clearly that null is correct when the information is not present. A schema that requires every field on a document where some fields are genuinely missing is a schema that requests fabrication.
For the others, validation beyond schema conformance is necessary — which is a separate layer.
Handling Uncertainty in the Schema
The most useful structural pattern is giving the model a way to express that it could not do the task.
{
"extraction_status": {
"type": "string",
"enum": ["complete", "partial", "not_found", "document_unreadable"]
},
"fields": { "...": "the actual extracted data, nullable throughout" },
"missing_fields": {
"type": "array",
"items": { "type": "string" },
"description": "Names of requested fields not present in the source"
}
}This changes the failure mode from silent fabrication to explicit reporting. A model that can say “this field was not present” will say so; a model with only a required string field will fill it.
Two related patterns are worth adopting. Requiring a source quote alongside each extracted value makes verification possible and measurably reduces fabrication, because the model must locate actual text. And a per-field confidence score, while imperfectly calibrated, is usable for routing — low-confidence extractions to human review, high-confidence ones through automatically.
Retry Strategy That Converges
Even with constrained decoding, requests fail — timeouts, truncation, semantic validation failures. Retry logic should improve the odds rather than repeat the same attempt.
Include the error in the retry. Sending the model its invalid output and the specific validation error resolves a large share of failures on the second attempt. Retrying blindly repeats the same conditions.
Reduce temperature on retry. If the first attempt was sampled at 0.7, retry at 0.2. Lower temperature produces more conservative, format-adherent output.
Simplify on the second retry. If a complex schema fails twice, requesting fewer fields or splitting into separate calls frequently succeeds where repetition does not.
Cap retries at two or three. Beyond that, failures are usually structural — the document does not contain what you are asking for, or the schema is wrong for this input. Continued retrying burns cost without converging.
Record what needed retries. Inputs requiring retries are your hardest cases and belong in your evaluation set. This is where the highest-value test data comes from.
Validation Layers
Four layers, each catching what the previous one cannot:
| Layer | Catches | Cost |
|---|---|---|
| Constrained decoding | Syntax errors | Free where supported |
| Schema validation | Type and shape violations | Negligible |
| Business rule validation | Impossible values, failed cross-checks | Low |
| Sampled human review | Plausible but wrong values | Ongoing |
The third layer is where most real errors are caught and it is the one most frequently omitted. Business rules encode what the schema cannot: a line-item total should sum to the invoice total, an end date should follow a start date, a quantity should be positive, a postal code should exist. These checks are cheap, deterministic, and catch exactly the type-valid nonsense that schema validation permits.
The fourth layer is the only way to detect errors that pass every automated check. A sampled audit of a few hundred outputs monthly, reviewed properly, is what tells you your actual error rate — which is otherwise unknown.
Common Pitfalls
Relying on prompting where constrained decoding is available. A 3 percent failure rate is unnecessary when a guarantee is offered.
Requiring fields that may legitimately be absent. This does not produce completeness; it produces fabrication.
Schema validation as the only validation. Type-correct nonsense passes. Business rules are where real errors surface.
Placing conclusion fields before evidence fields. Forces commitment before reasoning and reduces accuracy.
Blind retries. Feed the error back or nothing changes.
Deep nesting. Flatter schemas produce measurably fewer errors.
No sampled human audit. Without it, your error rate is unmeasured rather than low.
Conclusion
Reliable structured output is a solved problem where constrained decoding is available and a probabilistic one where it is not. If your provider supports schema-enforced generation, that single change eliminates the entire class of syntax failures.
What remains is semantic correctness, and the design choices that improve it are specific: descriptive field names and descriptions, enums for categories, nullable fields with explicit permission to return null, shallow nesting, evidence and reasoning fields placed before conclusions, and an explicit status field so the model can report failure rather than inventing values.
Then validate in layers. Schema conformance is the floor. Business rules catch the type-valid nonsense. And a sampled human audit is the only thing that tells you what your error rate actually is.
Frequently Asked Questions
Is constrained decoding available everywhere? Most major providers now offer some form of JSON mode or schema enforcement, with varying schema feature support. Self-hosted models can use grammar-constrained decoding libraries. Coverage is good and not universal.
Does constraining output reduce response quality? Sometimes, particularly for tasks needing extended reasoning. Including a free-text reasoning field before the structured fields recovers most of the loss while retaining the structural guarantee.
Should temperature be zero for structured output? Low, generally — 0 to 0.2. Structured extraction rarely benefits from sampling variety, and lower temperature improves both consistency and format adherence.
How should optional fields be represented? Nullable with a description stating that null is correct when the information is absent. Omitting the field entirely from the schema is also valid; the important part is that absence is expressible.
What about very large outputs? Split into several calls. A single request producing hundreds of records risks truncation and is harder to validate. Batching by section or page is more reliable and easier to retry granularly.
Can I trust confidence scores the model produces? For relative ranking, reasonably. For absolute calibration, no — a stated 0.9 does not mean 90 percent accuracy. They are useful for routing decisions and should not be treated as probabilities.
How do I test structured output reliably? Assert schema conformance on every case, then assert business rules, then compare extracted values against known-correct references on a labelled subset. Automate the first two in CI; run the third against a held-out set.
