JSON-Pass vs Value-Accuracy Gap

The single most consequential finding from the Structured Output Benchmark is the gap between JSON Pass Rate (the response is valid JSON) and Value Accuracy (the leaf values are exactly correct). Every frontier model clears 95%+ on JSON Pass; Value Accuracy sits 15-30 percentage points below.

The numbers

The tightest gap — Qwen3.5-35B at 16.8 pp — still means roughly 1 in 6 fields is wrong despite the JSON parsing cleanly. The widest gap, Schematron-8B at 25.6 pp, means a quarter of fields are wrong on schema-clean output.

Why JSON Pass stopped being a useful metric

Around 2023, frontier models routinely failed JSON parsing. Constrained-decoding tools (Outlines, JSON Mode, Structured Outputs) closed that gap. By 2026 it's a solved problem at the parse level — every API offers a "this WILL be valid JSON" mode, and the underlying models have learned schema syntax well enough that the pass rate clears 95%+ even without the constraint.

What constrained decoding can't fix is value correctness. A schema enforcer can guarantee the response has a total: number field. It cannot guarantee that total matches what the source document said.

Why the gap hides

Production teams reviewing structured-output performance often look at three signals:

1. Schema-validation rate — does the JSON parse and validate?
2. Field-presence rate — are all required keys there?
3. Type-correctness — are values the right primitive type?

All three can read 99%+ while value accuracy sits at 75%. Type Safety / Path Recall / Structure Coverage all cluster near the ceiling on SOB even when 20-30% of leaf values are wrong.

This produces the worst kind of false signal: the dashboard says "structured output is excellent," the production logs report no parse errors, and downstream systems silently accept hallucinated values until somebody notices an invoice_total is wrong by an order of magnitude.

What "structured hallucination" looks like

The hardest failure mode the SOB report calls out is the structured hallucination — a type-correct, schema-valid, plausible-looking value that's just wrong. SOB's example: ground truth "target_market_age": "15 to 35 years", the model returns "25 to 35". The output passes schema validation, the type is right, the format is right. Only an exact leaf-value comparison against verified ground truth catches it.

Structured hallucinations are particularly bad because the usual guardrails — schema enforcement, type checking, regex validators — are useless against them. They require either field-level human review, ensemble-and-disagree, or grounding-faithfulness checks against the source context.

Practical implications

If you're shipping LLM structured output to production and the SLA is "downstream systems trust the values without review":

• Don't treat schema validation as quality. Track exact-value accuracy against a held-out gold set.
• Don't extrapolate text benchmarks to image/audio. SOB's audio Value Accuracy peaks at 23.7% — meeting transcripts are out of reach for any current model without a different pipeline.
• The model with the highest JSON Pass isn't necessarily the best. Schematron-8B has the highest Path Recall but the lowest Value Accuracy. Picking by parse rate flips you to a worse model.
• Constrained decoding closes parse, not value. It's necessary but not sufficient.

Related

• structured-output-benchmark — the SOB benchmark this finding comes from
• interfaze — Interfaze entity
• average-is-all-you-need — adjacent argument about LLMs raising floors on average output while leaving the ceiling untouched