Asking an LLM to “return valid JSON” is not enough when your application depends on a specific structure. Define the data contract, use a provider’s schema-constrained output feature where it fits, then parse and validate the result—including business rules—in your own code. Structured output can reduce formatting failures; it cannot make the whole application reliable by itself.
Valid JSON is not the same as valid application data
JSON mode and schema-constrained output solve different problems. JSON mode aims to produce syntactically valid JSON; it does not necessarily ensure that an object has the required fields, types, or allowed values. OpenAI makes this distinction explicitly in its Structured Outputs guide: only Structured Outputs enforces adherence to supported schemas.
As an Amazon Associate I earn from qualifying purchases.
Even schema adherence is not semantic correctness. An object can have the right shape while containing a nonexistent identifier, an unsupported claim, or a value that violates a business rule. Treat the model response as untrusted input at the application boundary.
Define the contract before writing the prompt
Write a JSON Schema or equivalent typed structure that describes what the application can actually accept. Make the following explicit:
#1 Best Overall
- Which properties are required and which may be absent.
- The expected type for each value, including arrays and nested objects.
- Allowed values for enumerations and whether extra properties are permitted.
- Any constraints the chosen provider supports, such as formats or numeric limits.
Descriptions in the schema or prompt can clarify what a field means, but they do not replace validation. Keep rules that are hard to express in the schema—such as relationships between fields, authorization, or whether a referenced record exists—in application code.
Choose the generation interface for the job
For a user-facing response that must have a defined shape, use a structured response-format feature when the target model and API support the schema you need. If the model must request an application action, use tool or function calling with a strict schema where available. These are different interfaces: formatting a response as structured data does not grant permission to execute an action.
| Option | What it is for | Important qualification |
|---|---|---|
| JSON mode | Encourages syntactically valid JSON. | It does not guarantee conformance to a particular schema, and clients must account for incomplete-output edge cases, as described in the OpenAI guide. |
| Structured response format | Shapes a model’s user-facing response to a supported schema. | Availability and supported schema features depend on the provider, model, and API path. OpenAI recommends this over JSON mode when supported in its guide. |
| Tool or function calling | Lets a model request that the application invoke a function using structured arguments. | It is for connecting the model to application behavior, not merely formatting a user-facing response. Check the provider’s strictness options and limitations. |
OpenAI
OpenAI distinguishes JSON mode from Structured Outputs and says the latter adheres to supported schemas. Its interface choice should reflect whether you need a structured response or a function call; consult the current guide for model and API support, schema constraints, and incomplete-output handling.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Rank #2
Google Gemini
Gemini structured output supports a subset of JSON Schema, not every possible schema. Google warns that very large or deeply nested schemas may be rejected and recommends validating output in application code because schema-shaped results can still be semantically wrong. See Google’s structured-output documentation.
Anthropic Claude
Anthropic documents JSON outputs through output_config.format and separately documents strict tool use. Schema limits and model availability are provider-specific, so check the current documentation for the model and API path you deploy.
Do not assume the same schema or API parameters work across vendors. Before committing to an interface, check model availability, supported schema keywords and nesting, response-format versus tool-call semantics, refusal and interruption behavior, and how the SDK exposes parse errors.
Validate the response at your application boundary
- Check the provider result. Distinguish a completed response from a refusal, interruption, or API-level error using the status and fields exposed by the provider’s SDK.
- Parse and validate. Parse the returned content and validate the resulting object against the contract your application expects. Do not assume the SDK’s convenience parser checks every invariant you care about.
- Apply domain rules. Check ranges, cross-field relationships, identifier existence, authorization, and any other rule that determines whether a value is safe and usable.
- Only then use the data. Keep model output out of consequential application behavior until the relevant checks pass.
Google’s documentation puts the boundary plainly: “Always validate the final output in your application code before using it.” See Google’s structured-output documentation.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Give each failure a deliberate response
Not all failures call for the same retry. Identify the failure category first:
- Unsupported or overly complex schema: The API may reject the request before generation. Simplify or adapt the schema to the provider’s documented subset; repeatedly resending the identical request will not fix a deterministic incompatibility.
- Transport error, rate limit, or timeout: Apply the retry policy appropriate to the particular transient failure, respecting provider guidance and your latency budget.
- Incomplete or interrupted generation: Detect the incomplete result rather than trying to parse it as a complete object. Decide whether to retry, request a continuation if the interface supports it, or return a controlled failure.
- Explicit refusal: Handle it as a refusal, not as malformed JSON to repair automatically.
- Parse or schema-validation failure: This can occur when the selected mode does not constrain output, or at client boundaries. A bounded repair attempt may be appropriate for some workloads; do not loop indefinitely.
- Schema-valid but semantically invalid data: Reject or route it through the application’s domain-specific recovery path. Retrying the same prompt is not a substitute for checking business rules.
Log the failure category and enough diagnostic context to investigate, while avoiding unnecessary storage of sensitive prompts or outputs.
Test more than whether the output parses
A parse-success metric can look excellent while the model returns the wrong value in a perfectly valid object. Test the full contract with representative and adversarial inputs, including missing information, boundary values, refusal-triggering requests, long outputs, and schema features close to the provider’s documented limits.
Track at least parse success, schema compliance, semantic or business-rule validity, refusal and interruption rates, and end-to-end task success. Constrained-generation research treats schema coverage, efficiency, and output quality as distinct evaluation axes: the JSONSchemaBench paper describes a benchmark using 10,000 real-world JSON schemas. That is a benchmark scope, not a guarantee about a particular production workload.
Recommended Free Tools
Read reliability figures in their proper scope
OpenAI reported that gpt-4o-2024-08-06 achieved 100% on its complex JSON Schema-following evaluation with Structured Outputs, compared with less than 40% for gpt-4-0613. The same announcement says the newer model reached 93% on the stated benchmark before a deterministic constrained-decoding layer was added; OpenAI said the model’s nondeterministic behavior still fell short of developer reliability needs. These are results from OpenAI’s own evaluation of named models and a particular task, not a cross-provider comparison or a promise of semantic accuracy. See OpenAI’s August 6, 2024 announcement.
Best Value
The announcement also describes schema preprocessing and a first-request latency penalty for OpenAI’s implementation. Treat that as specific to the described implementation, not a general latency comparison. The available sources do not establish an apples-to-apples current comparison of provider latency or price.
Make the interface part of the system design
For a production integration, evaluate the exact model, API path, and schema you intend to deploy. Confirm supported keywords and complexity limits; test refusal, truncation, and error surfaces; and decide how the SDK parser fits into your own validation boundary. Then measure the workload that matters to your application. Provider feature names alone do not establish portability or end-to-end reliability.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
