You can make an AI system return data that conforms to a schema; you cannot guarantee from that fact alone that the values are true or supported by the document. Reliable extraction is an engineering process: define how to handle unknowns, preserve evidence for each value, validate the output, and measure field-level errors on representative documents. “Can’t hallucinate” is a design objective, not a capability that JSON formatting can guarantee.
Why valid structured output can still be wrong
A schema can require a particular shape—for example, an object with a date, a supplier name, and an amount. Constrained output can prevent malformed JSON or disallowed types, depending on the implementation and the schema features it supports. It cannot establish that the date or amount appears in the source, or that the model interpreted it correctly.
As an Amazon Associate I earn from qualifying purchases.
| Control or check | What it can establish | What it cannot establish by itself |
|---|---|---|
| JSON parsing | The output is syntactically valid JSON. | That a value is present in, or correctly interpreted from, the document. |
| Schema validation | Keys, types, required fields, and allowed values meet the configured rules. | That a schema-compliant value is factually supported. |
| Evidence attached to a field | Where to look in the document to audit a value. | That the evidence actually supports the value; the evidence itself must be checked. |
| Field-level evaluation against reviewed records | How the system performs on the tested fields and documents under the chosen scoring rules. | Perfect performance on new documents, layouts, schemas, or domains. |
The StructHallu-Drift study explicitly separates syntactic validity from semantic fidelity. OpenAI’s API documentation also describes schema adherence in strict mode while noting that strict mode supports a subset of JSON Schema. Both distinctions matter: structural guarantees apply only within the documented feature set, and neither syntax nor schema compliance is a factuality check.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
What evaluations show—and what they do not
Published results demonstrate why a format guarantee should not be presented as a guarantee against unsupported values. They are findings for particular benchmarks and tasks, not expected error rates for every deployed extraction system.
#1 Best Overall
| Study and task | Reported result | How to interpret it |
|---|---|---|
| ExtractBench (2026), PDF-to-JSON evaluation | The benchmark paired 35 PDF documents with schemas and human-annotated labels, producing 12,867 evaluatable fields. Validity fell to 0% for a 369-field financial-reporting schema across the models tested. | This is an extreme result for a very broad schema in that benchmark, not evidence that smaller schemas generally fail. |
| StructHallu-Drift (Mujtaba Hasan, ACL SURGeLLM workshop proceedings, July 2026) | Across 1,200 schema–model evaluation instances, 39–54% of structured outputs contained at least one semantic hallucination. | The result concerns the study’s evaluation instances; it is not a universal rate for production systems. |
| Chemistry procedure extraction (Royal Society of Chemistry, 2024) | After heuristic repair, 9,963 of 10,000 outputs were valid ORD records (99.6%). Under the study’s strict measure, ProductCompound message accuracy was 71.3%. | Near-perfect record validity did not mean uniformly accurate chemistry values. The study attributed many errors to implicit details, including calculated yields; these figures apply to its domain and setup. |
Together, these results show why validity and correctness need separate measurements. They do not establish one accuracy figure that applies across models, document types, schemas, or fields.
Design the schema to allow honest uncertainty
Include only information the workflow needs
Every additional field creates another opportunity for omission, misinterpretation, or unsupported completion, and nested arrays can make both extraction and evaluation more complex. Start with the fields downstream users actually need. Add breadth only when the use case justifies it, then test the expanded schema rather than assuming performance will hold.
Define what happens when the source is unclear
For each field, specify what to return when information is absent, ambiguous, or not stated. Depending on the schema and the consuming application, that may be null, an explicit status such as unknown, or omission. Distinguish “not stated” from “not applicable” if users or downstream systems need to treat them differently. Do not require the model to guess simply to populate every field.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Keep values traceable to source evidence
Ask the extractor to return a supporting span, page, table, or other document location for each value where practical. For example, a record can pair each extracted value with an evidence location and a status indicating whether it was found. Treat that trace as an audit aid, not proof: a model can produce a plausible-looking citation that does not support its answer, so the source passage still needs to be checked.
A practical extraction and evaluation workflow
- Specify fields and uncertainty behavior. Document each field’s meaning, type, allowed values, and whether it is required. Define the output for absent or ambiguous information before running the extraction.
- Constrain output to the schema. Use a structured-output feature or equivalent mechanism that supports the schema features you need. Check the provider’s current documentation for strict-mode limits rather than assuming all JSON Schema features are enforced.
- Validate deterministically. Parse the result and check required keys, types, allowed values, and any domain rules your application can test. Reject or route invalid records for correction. Passing validation means the structure meets those checks; it does not confirm source support.
- Check evidence against the document. Review whether each cited span, page, or table supports the value. Include checks for inferred or calculated values when the task is supposed to extract only what the source states.
- Build a human-checked reference set. Use documents representative of actual work, including difficult layouts, tables, nested structures, and fields that may be implicit. Have qualified reviewers establish the reference values and how ambiguous cases should be scored.
- Score errors by field and type. Separate omissions, unsupported additions, and incorrect values. Choose comparisons appropriate to the field: exact matching for identifiers, numeric tolerances where justified, and carefully defined semantic comparison where wording can vary.
- Compare configurations on identical tests. Hold documents, schemas, reference labels, and scoring rules constant when comparing models or settings. Re-run evaluation after material schema changes, and involve domain experts when errors could have significant consequences.
Choose evaluation measures that reveal the failure
A single pass/fail validity rate hides important differences. FAIRmat-NFDI’s JSON Extract Eval supports field-specific comparators and reports precision, recall, F1, omissions, hallucinations, and mismatches. JSONSchemaBench evaluates constrained decoding across constraint compliance, schema coverage, and output quality. These frameworks illustrate complementary questions: did the output obey the structure, did the system cover the schema, and were the extracted values right?
Choose measures that match the cost of each error. For a high-risk field, an unsupported value may be more serious than a blank one; for another workflow, missing a required value may be the principal concern. Report the field-level results and scoring rules so a strong aggregate score does not conceal a weak or safety-critical field.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How to compare extraction options
When assessing a schema-constrained API, document-processing platform, or evaluation framework, test it on the same target documents and schema rather than relying on a general claim of “structured output.” Compare:
Quick Recap
- Structural compliance: which schema features are supported and how invalid outputs are handled.
- Semantic performance: field-level accuracy on your document types, including omissions, unsupported values, and mismatches.
- Uncertainty handling: whether absent and ambiguous values can be represented without forcing a guess.
- Auditability: whether values can be tied to source spans, pages, tables, or other useful locations.
- Hard cases: wide schemas, nested arrays, scans, tables, and fields that require interpretation rather than direct copying.
- Operational fit: privacy terms, throughput, cost, and human-review requirements. These vary by provider and need to be checked in current provider documentation.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
