Any system that consumes model output programmatically needs that output to have a shape. The naive approach — asking for JSON in the prompt — works often enough to reach production and fails often enough to page someone. The gap between 'usually valid' and 'always valid' is where the engineering is.
How output gets constrained
- Prompt instruction. Ask for JSON. Cheapest, weakest — the model may add prose, wrap it in a code fence, or omit a field.
- JSON mode. The provider guarantees syntactically valid JSON. It does not guarantee your schema.
- Schema-constrained decoding. The provider restricts token sampling so only outputs conforming to your schema are possible. This is the one that turns a hope into a guarantee.
- Validate and retry. Parse against your schema; on failure, retry with the error. The safety net under whichever of the above you use.
Design the schema for the model, not just the parser
{
"type": "object",
"properties": {
"sentiment": { "enum": ["positive", "neutral", "negative"] },
"confidence": { "enum": ["high", "medium", "low"] },
"reason": { "type": "string", "maxLength": 200,
"description": "One sentence quoting the phrase that decided it." },
"needs_human_review": { "type": "boolean" }
},
"required": ["sentiment", "confidence", "needs_human_review"],
"additionalProperties": false
}- Enums over free strings. A closed set removes a whole category of downstream normalisation.
- Enums over numeric confidence. A model asked for 0-1 will emit 0.85 with no calibration behind it; three named levels are more honest and more useful.
- `additionalProperties: false` or you will discover invented fields in production.
- Describe every field. The description is prompt text and materially changes accuracy.
- Include an escape hatch — a `needs_human_review` flag or a nullable field — so the model has a way to express uncertainty other than fabricating.
Valid does not mean correct
Schema constraint guarantees shape, not truth. A response conforming perfectly to your schema can still assert the wrong sentiment, cite a non-existent order, or fill a required field with a plausible invention. Validation is a parsing control, never a correctness control — the correctness check is a separate evaluation.
Put reasoning before the answer
Field order in a schema is generation order. Placing a short reasoning field before the conclusion lets the model condition its answer on its own reasoning; placing it after produces a justification for an answer already committed to. Same fields, measurably different accuracy.
Handle the failures you will still get
- Validate every response, including when the provider guarantees the schema — providers have bugs and versions change.
- Retry once with the validation error included, then stop. A model that failed twice is unlikely to succeed on the fifth attempt and you are paying for each.
- Log the raw output on failure. The malformed response is the only evidence of what went wrong.
- Decide what a permanent failure means for the caller — a null result and a clear error beats a silently empty object.
Frequently Asked Questions
Does constrained decoding hurt quality?
It can, if the schema forces a shape the model would not naturally produce. Overly deep nesting and long required-field lists are the usual culprits - flatter schemas tend to perform better.
Should I use JSON or a simpler format?
JSON where you have schema-constrained decoding available. For simple extraction, a delimited or line-based format is sometimes more reliable and much cheaper in tokens.
How do I version a schema?
As you would an API contract: additive changes only where possible, and if a consumer depends on the shape, that dependency needs the same care as any other interface.
References
- JSON Schema Specification — JSON Schema
- Context Engineering: A Practical Guide for AI Agents (2026) — Sourcegraph
About Jishu Labs
Jishu Labs is a software development company founded in 2016. We build custom software, AI/ML systems, and full-stack web and mobile applications for clients, and we make eight AI tools for software teams.