# An Empirical Study on Strong-Weak Model Collaboration for Repo-level Code Generation

A May 2025 preprint from Carnegie Mellon's Language Technologies Institute (Shubham Gandhi, Atharva Naik, Yiqing Xie, Carolyn Rosé). The question is economic: if a strong model resolves more GitHub issues but costs far more, which ways of splitting the work between a strong and a weak model get close to the strong model's results at a fraction of the price?

## Setup

Everything runs inside Agentless Lite, a two-step retrieval-augmented pipeline. voyage-code-3 retrieves the top five relevant files, then a model generates SEARCH/REPLACE patches, retrying up to ten times with the temperature raised by 0.1 after each invalid patch. The benchmark is SWE-bench Lite, 300 issues from 11 Python repositories, each run once. Strong models are o3-mini, o4-mini and GPT-4o-mini; weak models are GPT-4o-mini and Qwen2.5-Coder at 7B, 14B and 32B. Cost is API spend for all 300 instances, with retrieval excluded because it is constant.

Alone, o4-mini resolves 45.33% for $33.33, o3-mini 33.67% for $46.22, GPT-4o-mini 15.33% for $4.53, and Qwen2.5-Coder-32B 20.33% for an estimated $7.02.

The strategies fall into four groups. Cost-equated weak-only baselines sample the weak model until it has spent what the strong one would, then pick a patch by majority vote, clustering, the weak model's own choice, or an oracle best-of-n. Static context augmentation has the strong model write something the weak one then uses: a repository summary, repository FAQs, a RepoGraph structure, few-shot examples, or per-issue plans and Q&A. Pipeline division runs the two in a fixed order: Strong LM First (the strong model makes the attempt and the weak model only iterates until the patch is well-formed), Weak LM First (a cascade where the weak model tries and the strong model gets one attempt only if no valid patch comes out), and Prompt Reduction (the weak model prunes the retrieved code before the strong one writes). Dynamic collaboration uses a router, either weak or strong, to label each issue simple or complex and send it to the matching model. The figure caption counts 14 techniques, the introduction says 12 methods, and the statistical appendix counts 19 configurations once variants are split out.

## Results

Strong LM First is the quality winner. With o4-mini and GPT-4o-mini it resolves 41.67% for $25.34, against 45.33% for $33.33 with o4-mini alone, and roughly 92% more than the best cost-equated weak baseline (oracle best-of-n at 21.67%). With o3-mini and Qwen2.5-Coder-32B it resolves 33.00% for $27.74 against o3-mini's 33.67% for $46.22, which is the "equivalent performance at 40% less cost" in the abstract. In the per-method ANOVA, Strong LM First and a single strong attempt were significantly ahead of everything else on accuracy.

Weak LM First is the budget option and its quality depends on how weak the weak model is. Paired with o4-mini, GPT-4o-mini under Weak LM First resolves 15.33% for $6.06, the same as GPT-4o-mini alone. Paired with o3-mini, Qwen2.5-Coder-7B goes from 4.67% to 19.33%, because the 7B model so often fails to produce a valid patch that the strong model gets called. The authors' guidance is that Weak LM First and the weak router suit tight budgets and Strong LM First suits larger ones, with the cost-performance curves crossing.

Spending the strong model's budget on more weak samples was the worst use of money. Self-consistency over about 15 GPT-4o-mini samples scored 14.33% to 16.33% at o4-mini-level cost, no better than one weak attempt. Repository-level context (summaries, structure, FAQs) did not help the weak model and few-shot examples often hurt, while per-issue plans from the strong model did: 29.33% for GPT-4o-mini with an o4-mini plan. A weak router often beat a strong one. The text reports six points higher at about 20% lower cost and attributes it to the strong model overthinking the routing decision; in the o3-mini plus Qwen2.5-Coder-32B table the weak router scores 29.00% for $32.77 against the strong router's 23.00% for $40.76 (the prose names o4-mini for that pair, which does not match any table). Prompt Reduction cut the valid-patch rate to about 65% but sometimes raised resolution, a high-variance trade.

## How it gets cited

The [[cross-model-code-review]] post that led to this ingest cites the paper in support of a weak model writing code and a strong model reviewing it. The paper does not test that arrangement. None of its strategies is a review stage, and in the one that performs best the roles run the other way: the strong model writes and the weak one only tidies formatting. The strategy where the weak model does most of the work is the cheap cascade, which raises quality little when the weak model is competent enough to produce valid patches. What the paper does support is the general claim that pairing models beats spending the same money on more samples from the cheap one.

## Limits

One simple framework rather than an agent, one run per instance, Python only, inference-time methods only, cost measured in API tokens without latency or energy, and models from mid-2025. The authors say the taxonomy should transfer; the numbers are specific to Agentless Lite and these pairs.

## Cross-references

- [[cross-model-code-review]] — what this and five other papers say about dividing writing and checking between models
- [[small-model-code-judges]] — a cheaper split where a small judge chooses among a small generator's candidates
- [[scaffold-model-fit]] — why a result measured inside one scaffold is a model-times-scaffold number
- [[local-ai-is-not-opus]] — the operator's version of the strong/weak cost trade