Structured Output Benchmark (SOB)

The Structured Output Benchmark (SOB) was published by Interfaze (with arXiv paper 2604.25359, HuggingFace dataset, and a hosted leaderboard) to give a single answer to "how well does this model produce structured data from messy real input." It's directly aimed at production use cases — invoice parsing, medical records, meeting transcript extraction, PDF-to-database — where a hallucinated value or a swapped field silently breaks downstream code.

What earlier benchmarks miss

Most existing structured-output benchmarks (JSONSchemaBench, StructEval, DeepJSONEval, LLMStructBench) collapse the question into "does the response parse and validate against the schema." SOB calls out five specific failure modes of that framing:

• A model can emit perfect JSON with completely wrong values and score 100%
• Single-source text-only inputs miss the realistic OCR / screenshot / audio cases
• Easy and hard schemas are scored the same, hiding which models actually handle nesting
• Parse errors, structure errors, and value errors all get blended into one number
• Reasoning / chain-of-thought leaks into the score, so you can't isolate the extraction skill

How SOB is built

Three modalities, one scoring pipeline. Records come from HotpotQA (5,000 text), olmOCR-bench (209 images), and the AMI Meeting Corpus (115 audio). Each record is paired with a JSON Schema and a verified ground-truth answer (human-authored, LLM cross-checked). To isolate structured-output skill from vision-OCR / ASR, image and audio records are pre-converted to text-normalized context — so all models see the same modality-stripped input and the differences are attributable to schema handling.

Seven metrics, not one. Per record:

Two scoring gates. Hardening: if JSON parse fails, downstream semantic metrics zero for that record. Coverage: Value Accuracy is only credited on fields the model returned, with missing paths counted as wrong. Schema complexity is weighted (easy=1.0, medium=2.0, hard=3.0) so the leaderboard reflects nested-schema handling, not just easy cases.

No reasoning. All models run at temperature 0.0, max 2048 output tokens, with reasoning/thinking turned off wherever the provider allows. The score reflects pure extraction capability.

The headline finding

The overall leaderboard top-five clusters within 1 point: GPT-5.4 (0.870), GLM-4.7 (0.861), Qwen3.5-35B (0.861), Gemini-2.5-Flash (0.860), Qwen3-235B (0.857). Interfaze's own beta sits at #6 (0.855), Claude-Sonnet-4.6 at #7 (0.854).

The headline isn't the ranking — it's the JSON-Pass-vs-Value-Accuracy gap:

Every modern model clears 95%+ on JSON Pass — that test stopped separating frontier models years ago. Value Accuracy sits 15-30 points lower across the board. A pass-rate-only benchmark has been hiding a third of the actual error rate in every published headline. See json-pass-value-accuracy-gap.

Patterns worth internalizing

1. Valid JSON ≠ correct JSON. The gap above.
2. Structural metrics mask value errors. Path Recall / Structure Coverage / Type Safety can all read 99%+ while 20-30% of leaf values are wrong, and Perfect Response collapses to about half even for the best models.
3. Model size is not a predictor. Qwen3.5-35B and GLM-4.7 beat GPT-5 and Claude-Sonnet-4.6 on Value Accuracy. Phi-4 (14B) edges out GPT-5 and GPT-5-Mini on text.
4. Structured hallucinations are the hardest bug. A type-correct, schema-valid, plausible-looking wrong value slips through all guardrails. SOB's example: ground truth "target_market_age": "15 to 35 years", model returns "25 to 35" — invisible without field-level checks.
5. Modalities don't transfer. Best Value Accuracy: text 83.0%, image 67.2%, audio 23.7%. Audio collapses because transcripts are long (~7,300 tokens) and have overlapping speakers.
6. Per-modality leaders differ. GLM-4.7 leads text, Gemma-4-31B leads images, Gemini-2.5-Flash leads audio. No single model dominates all three. Text-only leaderboards mask this.

What SOB doesn't claim

The eval is single-run, low temperature, no reasoning. So it doesn't characterize variance, doesn't show how reasoning models behave when reasoning is on, and doesn't cover recursive types, unions, or large enum spaces. The team flags those as next-version axes.

The benchmark also doesn't tell you which model is "best" — rank order is metric-specific. The report is structured to push readers toward picking the metric that matches their production constraint (Value Accuracy if a downstream pipeline trusts fields without review, Faithfulness if hallucinations would be costly, Perfect Response if the whole record must be right).

Why Interfaze published

Interfaze is building a structured-output model and sells the benchmark partly as positioning for its own product (which sits at #6 on the unified leaderboard). The framing is honest about that: "the first step to making structured output better is to measure it and hold ourselves against the best." The benchmark, dataset, and evaluation harness are open (GitHub).

Related

• interfaze — entity
• json-pass-value-accuracy-gap — concept page on the headline finding
• clean-code-coding-agents — same family of "the surface metric isn't the right one" arguments, different domain
• claude-emotion-concepts — adjacent: another open eval where the methodology is more interesting than the ranking