EnglishРусский Map
↑Cross-model code review

Variation in Verification: Understanding Verification Dynamics in LLMs

Wiki summaryllmllm-evaluationverificationbenchmarks ↳ show in map Markdown
title
Variation in Verification: Understanding Verification Dynamics in LLMs
type
summary
summary
Salesforce study finding errors from stronger generators are harder for any verifier to catch, so a weak generator plus GPT-4o nearly matches a strong one
parent
cross-model-code-review
tags
llm, llm-evaluation, verification, benchmarks
sources
arxiv-2509-17995, arxiv-2509-17995-full
created
2026-09-13
updated
2026-09-13

A study by Yefan Zhou, Austin Xu, Yilun Zhou, Janvijay Singh, Jiang Gui and Shafiq Joty (Salesforce AI Research, with Dartmouth and UIUC), first posted September 2025 and revised April 2026. The common practice it questions is defaulting to a frontier model as the verifier on the assumption that verification quality scales with the verifier's own ability to solve the problem. The paper asks what else decides whether a verifier gets it right.

Setup

The verifiers are generative: the model writes a chain of thought and ends with a Correct or Incorrect verdict, without a reference answer. Fourteen open models from 2B to 72B (Qwen2.5, Qwen3, Llama 3.x, Gemma-2, Ministral and Mistral-Small) and GPT-4o each act as both generator and verifier. Problems come from 12 benchmarks with objective answers: 2,347 math problems (a GSM8K subset, MATH500, OlympiadBench, AIME24/25, AMC23, Minerva-Math, BBEH multi-step arithmetic), 1,196 MMLU-Pro questions, and 901 natural-language reasoning problems (ReClor, FOLIO, GPQA Diamond). Every model samples 64 responses per problem. A problem's difficulty is the average pass rate across all generators. Verifiers are scored by true positive rate (accepting correct answers) and true negative rate (rejecting incorrect ones).

Three findings

Problem difficulty mainly decides whether correct answers get accepted. True positive rate rises steadily as problems get easier, while true negative rate shows no clear trend. The mechanism is that verifiers tend to re-solve the problem and compare; on hard problems their own solution is wrong more often, and 39.1% of false negatives on the hard set contain a mistake in the verifier's reference answer.

Generator capability mainly decides whether errors get caught. As the generator gets stronger, true negative rate falls substantially for nearly every verifier and domain. Qwen2.5-72B as a math verifier rejects 68% of wrong answers from Llama-3.1-8B and 17% of wrong answers from Qwen3-32B. Weak generators make surface errors such as self-contradictions, which verifiers reject; strong generators produce internally consistent chains where an early mistake, a missed case for instance, propagates cleanly into a well-structured wrong answer. An LLM-as-judge pass over generator outputs confirms that surface-level errors become rarer as capability rises.

Verifier capability correlates with verification accuracy overall, but the shape depends on difficulty. On medium problems the relationship is close to linear (r above 0.9). On easy problems there is a threshold near the top of the capability range. On hard problems more capability barely helps: math plateaus around 0.65 balanced accuracy, and on hard natural-language reasoning verifiers score below random with no meaningful relationship at all. The findings are reported to hold for reasoning models, for Qwen3-235B, and across verification prompts.

Test-time scaling

The applied half filters the 64 samples through a verifier and reports the pass rate of what survives. With GPT-4o as a fixed verifier on 181 math problems of difficulty 0.7–0.8, the gap between Gemma2-9B and Gemma2-27B shrinks from 10.3 points to 2.5, closing 75.7% of it. Across whole domains, most weak-strong generator gaps shrink by 30–50%. Verification gain peaks for weak-to-medium generators, which make errors a verifier can reject while producing enough correct answers to keep.

Swapping GPT-4o for Qwen2.5-7B as the verifier costs little in three regimes: very easy problems, very hard problems, and strong generators. The authors are explicit that these are the regimes where verification gains are 0.1 or less for both verifiers, so the two converge because neither helps. Their phrasing is that scaling the verifier alone does not overcome the underlying difficulty.

An appendix case study addresses cost. On the same 181 problems, Qwen3-4B alone scores 0.938, GPT-4o as generator 0.952, and Qwen3-4B generating with GPT-4o verifying 0.954. GPT-4o uses 193 tokens per prompt as verifier against 482 as generator (the prose says 483), about 2.5 times fewer, because verdicts are short.

Limits

Every domain has checkable ground truth, and none is code. Verification here filters a pool of samples; there is no critique handed back and no revision, so the paper measures detection, not repair. Difficulty is an oracle measure built from ground-truth labels, although a label-free estimator is reported to give the same trends. The related-work section cites concurrent work (Lu et al., 2025) finding cross-family verification particularly effective; that paper is not in the vault.

Cross-references

Linked from