# Reviewer Capability Governs Rejection Targeting, Not Repair Skill

A single-author preprint by Faizan Tanveer (FAST NUCES, Lahore), posted 2 September 2026. Multi-agent pipelines often put a cheaper model in the reviewer role to save money, and earlier studies showed verification stages do not always help, but they held the reviewer at roughly the executor's level. This one holds the executor fixed and swaps reviewers across a capability range, down to a model that cannot solve the problems at all, then classifies what every single rejection did. The author calls it a controlled pilot and not a general claim, and the page follows that framing.

## Setup

The pipeline has two stages. An executor writes a solution, a reviewer accepts or rejects it with a written critique (without seeing the reference answer), and on rejection the executor revises once. The executor is Gemini 3.1 Flash-Lite, which solves 52% of the problem set unaided. Four conditions vary the reviewer: no review; self-review by the same model; gpt-oss-20b as a cross-family mid-tier reviewer; and Llama-3.1-8B as a weak reviewer that solves approximately none of the problems. All three models are from different families, so self-review and cross-family review are not confounded with capability.

The problems are 100 Omni-MATH olympiad questions in difficulty band 5.0–6.0, restricted to numeric answers so grading is exact and no model sits in the scoring path. The design is paired: the executor solves each problem once and that same first answer is reviewed under every condition, so differences come from the reviewer and not from sampling. A provider rate limit forced the reviewer's input to be cut to the solution's final 2,000 characters, and even then 16% of the cross-family reviews came back unparseable and were counted as accepts, which makes that reviewer's measured recall a lower bound.

## Results

The cross-family reviewer raised accuracy from 52% to 64% (exact McNemar, p = 0.0005). Twelve problems went from wrong to right and none went the other way.

Self-review had the highest error-detection rate of any condition, a recall of 0.85, and produced no significant gain: +6 points, not significant at this sample size. It got its recall by rejecting a lot. It rejected 2.1 times as often as the cross-family reviewer and falsely rejected 35% of answers that were already correct, against 2% for the cross-family reviewer (paired p = 0.000015). Only 15% of its rejections repaired a wrong answer, against 43% (p = 0.0074), and 41% of its rejections changed nothing, against 14%. In selectivity terms, self-review needed 6.6 rejections for each useful repair and the cross-family reviewer 2.3. The direct accuracy comparison between those two conditions (p = 0.238) is underpowered, and the paper does not claim it.

Self-review's low damage rate turns out to be luck on the executor's side. Of its 18 false rejections, the executor went along with the critique 3 times and all 3 answers became wrong; in the other 15 it returned the same correct answer. Had it complied with every false rejection, accuracy would have fallen below the no-review baseline. The author calls this revision inertia: a conservative executor can make an incompetent reviewer look safe.

The weak reviewer changed none of the 100 final answers and roughly doubled token cost. Under the shortened prompt it wrote fluent, hedged prose and accepted almost everything; under the original longer prompt it had mostly returned empty text. That makes the capability floor prompt-conditional, as the paper says, and not a fixed property of the model.

The title's distinction comes from separating targeting from repair. Over the 52 correct first answers, 17 were falsely rejected by self-review and not by the cross-family reviewer, and zero the other way round. Repair skill, conditioned on the executor actually changing its answer, was 26% against 50% (p = 0.096, not significant with 35 and 24 attempts), and miscorrection rates were close at 39% and 43%. Reviewer capability decided which answers got rejected far more clearly than it decided how well a rejection was fixed.

Token cost per additional correct answer was about 24,000 for the cross-family reviewer and about 26,000 for self-review, so the better reviewer was not cheaper per gain. It bought twice the improvement with no damaged answers.

## The transcripts

Three failure modes are illustrated with hand-picked examples. Self-review fabricated justifications: on one geometry problem it invoked the nine-point circle to argue a count of 17 was too high, the executor complied, and the reference answer 17 became 7. It also asserted formulas without support, one of which yields 42 where the answer is 38. The weak reviewer produced articulate endorsements of wrong answers by restating the executor's reasoning in polished language. The author's point is that a critique can be specific, formatted and mathematical while verifying nothing.

## Limits and provenance

One executor, one benchmark, one difficulty band, one revision round, one prompt template, math and not code, and free-tier limits on model choice and sample size. Several proportions sit on small denominators (3 damage cases, 9 and 12 repairs), and pairwise tests are uncorrected for multiplicity. The pre-registered plan was a different study, whether cross-model error diversity predicts multi-agent benefit; calibration showed no difficulty band where all three models erred comparably, and the study was re-scoped to treat the capability spread itself as the variable, with the amendment written before the reported run. Both documents are kept in the repository.

The HTML conversion drops the results tables, so the reviewers' own unaided scores on the problem set are not in the vault copy. The paper calls gpt-oss-20b mid-tier and does not claim it is stronger than the executor, which matters when this result is cited for "a stronger model should review."

## Cross-references

- [[cross-model-code-review]] — the synthesis of this paper with the others on reviewer and writer roles
- [[llm-self-verification-limits]] — the 2024 GPT-4 study that found the same over-rejection by a self-critic
- [[verification-dynamics-llms]] — a larger study of how verifier and generator capability interact
- [[llm-output-variance]] — why a 100-item pilot should be read as a direction, not a size