LLM engineering · September 9, 2026
Structured Outputs: Valid JSON Is Not Correct JSON
Constrained decoding guarantees your schema parses. It guarantees nothing about the values inside it, and schema shape changes answer quality more than most teams realise.
By Shihab Shahriar Antor · Updated 2026-09-09
Getting structured data out of a language model used to mean asking for JSON, parsing it, and retrying when it came back wrapped in a code fence with a trailing comma. That problem is solved. Every major provider now offers some form of guaranteed schema adherence, and it works.
What has not changed is that the model still has to be right. Schema enforcement moved the failure from a parse exception you noticed to a well-formed record you did not.
Three mechanisms, often confused
| Mechanism | Guarantee | Use when |
|---|---|---|
| JSON mode | Output is syntactically valid JSON. No guarantee about which fields appear or what type they are. | Rarely the right choice now that schema-constrained modes exist. Useful for free-form JSON where you genuinely do not know the shape. |
| Tool or function calling | Arguments conform to the declared schema. The model also decides whether to call at all, which is a separate behaviour you have to handle. | The model is choosing an action. The schema is a side effect of describing the action, not the goal. |
| Constrained decoding, sometimes called strict mode | Output matches the schema, enforced during generation by masking tokens that would make it invalid. | You want data back and there is no decision to make. This is the default for extraction and classification. |
Constrained decoding works by restricting the token distribution at each step to tokens that keep the output a valid prefix of the grammar. It is a hard constraint, not a preference, which is why it cannot fail syntactically and why it can distort what the model wanted to say.
Schema shape changes answer quality
This is the part that surprises people. The schema is not a passive container. It sits in the prompt, it constrains generation order, and both of those affect what the model produces.
- 01
Put reasoning fields before answer fields
Generation is left to right and the model conditions on what it has already emitted. A field holding a short justification, declared before the field holding the verdict, gives the model tokens to think in. Declared after, it is a rationalisation of an answer already committed to, which is worth much less.
- 02
Prefer enums to free strings
An enum constrains decoding to the allowed values, which removes an entire class of near-miss output such as returning a lowercase variant or a synonym. It also makes the downstream mapping table unnecessary.
- 03
Give every field a description
Field descriptions are part of the prompt the model sees. An ambiguous field name with no description is a guess. This is the cheapest accuracy improvement available and it is routinely skipped.
- 04
Give the model a way to say the value is absent
Faced with a required field and no supporting evidence, a model fills it in. Provide an explicit null or an unknown enum member and say in the description when to use it, otherwise your absent values arrive as confident inventions.
- 05
Keep nesting shallow
Deeply nested objects degrade accuracy and increase the chance of a structurally valid but semantically scrambled result. Two or three levels is usually the practical limit; beyond that, run several extractions rather than one large one.
- 06
Split large schemas
A schema with forty fields makes every field slightly worse. Several focused calls usually beat one comprehensive call on accuracy, and they parallelise, which often makes them faster in wall-clock terms despite costing more tokens.
The JSON Schema you write is not the JSON Schema they run
Every provider supports a subset, and the subsets differ. Constructs that are commonly restricted or ignored include regular expression patterns, numeric minimum and maximum, string length bounds, union types, recursive references, and additional properties.
The dangerous case is not rejection. A rejected schema is a loud error you fix in a minute. The problem is a constraint that is accepted and then not enforced, so your maximum: 100 passes validation at request time and 340 arrives in production. Assume every value constraint is advisory until you have tested it, and re-validate everything after parsing regardless.
Refusals need a path
A strict schema and a safety refusal are in direct conflict: the model wants to emit an explanation, and the grammar only permits your object. Providers handle this differently, and some return a separate refusal field outside the schema. Whatever the mechanism, your code needs a branch for it. Systems that assume a schema-shaped response always arrives break in exactly the situation where breaking loudly matters.
Validation that still has to happen
Schema conformance covers types and required fields. It cannot cover anything that depends on meaning or on relationships between fields.
| Check | Example | Where it belongs |
|---|---|---|
| Cross-field consistency | An end date before the start date. Both are valid dates and the record is valid. | Application validation after parsing. |
| Groundedness | An extracted total that appears nowhere in the source document. | Verify against the source. For extraction, requiring a verbatim span alongside each value makes this checkable. |
| Referential integrity | A category identifier that is well-formed and does not exist in your database. | Lookup, then a defined fallback for the miss. |
| Numeric plausibility | A quantity three orders of magnitude outside anything you have seen. | Range checks derived from your own data, not from the schema. |
Tooling
Instructor maps provider structured output onto Pydantic models and handles retries and validation errors, which makes it the shortest path in Python. Outlines implements the constrained generation itself and is the right choice when you run your own models or need grammars beyond JSON. XGrammar is the fast grammar backend now used inside several serving stacks and is worth knowing about if constrained decoding overhead shows up in your latency numbers.
The rule that covers most of it
Treat a structured response as untrusted input that happens to have the right shape. It came from a probabilistic system, and the schema checked its grammar, not its truthfulness. Every validation you would write for a form submitted by a stranger still applies.
The evaluation side of this, including why a passing schema check is a floor rather than a score, is covered in testing LLM applications without fooling yourself.
Questions
- What are structured outputs in LLMs?
- A mode where the model is constrained to produce output matching a schema you supply, enforced during generation by masking tokens that would break validity. The result is guaranteed to parse and to contain the declared fields with the declared types. It is a syntactic guarantee only; the values it contains can still be incorrect.
- Do structured outputs reduce hallucination?
- They eliminate malformed output and they constrain values where you use enums, which removes some invention. They do not make the model more truthful about anything it is free to fill in. In one respect they make hallucination harder to notice, because the parse error that previously signalled confusion no longer occurs.
- Does the order of fields in a schema matter?
- Yes, and more than most people expect. The model generates fields in order and conditions each one on what it has already produced. A short reasoning field placed before the answer field measurably helps on judgement tasks; the same field after the answer contributes far less, because the answer was already committed.
- Why does the model invent a value instead of leaving a field empty?
- Because a required field with no null option gives it no way to express absence, and generation continues regardless. Add an explicit null or an unknown enum member and describe in the field description exactly when to use it. Without that, every missing value in your source arrives as a confident fabrication.
- Is JSON mode the same as structured outputs?
- No. JSON mode only guarantees the output is syntactically valid JSON, with no guarantee about fields or types. Structured outputs constrain generation to a specific schema. If you need particular fields back, JSON mode alone still requires you to validate and retry.
- Are all JSON Schema features supported?
- No, and support varies by provider. Regular expression patterns, numeric bounds, string length limits, unions and recursion are commonly restricted or silently unenforced. The risk is not the schema being rejected, which is obvious, but a constraint being accepted and ignored, so validate values in your own code after parsing.