# JSON-Pass vs Value-Accuracy Gap

The single most consequential finding from the [[structured-output-benchmark|Structured Output Benchmark]] is the gap between JSON Pass Rate (the response is valid JSON) and Value Accuracy (the leaf values are exactly correct). Every frontier model clears 95%+ on JSON Pass; Value Accuracy sits 15-30 percentage points below.

## The numbers

| Model | JSON Pass | Value Accuracy | Gap |
|---|---|---|---|
| GPT-5.4 | 99.3% | 79.8% | 19.5 pp |
| Qwen3.5-35B | 96.9% | 80.1% | 16.8 pp |
| GLM-4.7 | 96.5% | 80.4% | 16.1 pp |
| Gemini-2.5-Flash | 97.2% | 79.6% | 17.6 pp |
| Claude-Sonnet-4.6 | 97.9% | 77.9% | 20.0 pp |
| GPT-5 | 98.3% | 76.9% | 21.4 pp |
| Schematron-8B | 98.7% | 73.1% | **25.6 pp** |
| Nemotron-3-Nano-30B | 98.7% | 74.7% | 24.0 pp |
| Qwen3-30B | 98.3% | 75.3% | 23.0 pp |

The tightest gap — Qwen3.5-35B at 16.8 pp — still means roughly 1 in 6 fields is wrong despite the JSON parsing cleanly. The widest gap, Schematron-8B at 25.6 pp, means a quarter of fields are wrong on schema-clean output.

## Why JSON Pass stopped being a useful metric

Around 2023, frontier models routinely failed JSON parsing. Constrained-decoding tools (Outlines, JSON Mode, Structured Outputs) closed that gap. By 2026 it's a solved problem at the parse level — every API offers a "this WILL be valid JSON" mode, and the underlying models have learned schema syntax well enough that the pass rate clears 95%+ even without the constraint.

What constrained decoding *can't* fix is value correctness. A schema enforcer can guarantee the response has a `total: number` field. It cannot guarantee that `total` matches what the source document said.

## Why the gap hides

Production teams reviewing structured-output performance often look at three signals:

1. **Schema-validation rate** — does the JSON parse and validate?
2. **Field-presence rate** — are all required keys there?
3. **Type-correctness** — are values the right primitive type?

All three can read 99%+ while value accuracy sits at 75%. Type Safety / Path Recall / Structure Coverage all cluster near the ceiling on SOB even when 20-30% of leaf values are wrong.

This produces the worst kind of false signal: the dashboard says "structured output is excellent," the production logs report no parse errors, and downstream systems silently accept hallucinated values until somebody notices an `invoice_total` is wrong by an order of magnitude.

## What "structured hallucination" looks like

The hardest failure mode the SOB report calls out is the structured hallucination — a type-correct, schema-valid, plausible-looking value that's just wrong. SOB's example: ground truth `"target_market_age": "15 to 35 years"`, the model returns `"25 to 35"`. The output passes schema validation, the type is right, the format is right. Only an exact leaf-value comparison against verified ground truth catches it.

Structured hallucinations are particularly bad because the usual guardrails — schema enforcement, type checking, regex validators — are useless against them. They require either field-level human review, ensemble-and-disagree, or grounding-faithfulness checks against the source context.

## Practical implications

If you're shipping LLM structured output to production and the SLA is "downstream systems trust the values without review":

- **Don't treat schema validation as quality.** Track exact-value accuracy against a held-out gold set.
- **Don't extrapolate text benchmarks to image/audio.** SOB's audio Value Accuracy peaks at 23.7% — meeting transcripts are out of reach for any current model without a different pipeline.
- **The model with the highest JSON Pass isn't necessarily the best.** Schematron-8B has the highest Path Recall but the lowest Value Accuracy. Picking by parse rate flips you to a worse model.
- **Constrained decoding closes parse, not value.** It's necessary but not sufficient.

## Related

- [[structured-output-benchmark]] — the SOB benchmark this finding comes from
- [[interfaze]] — Interfaze entity
- [[average-is-all-you-need]] — adjacent argument about LLMs raising floors on average output while leaving the ceiling untouched