Improving Code Generation via Small Language Model-as-a-judge

An ICSE 2026 paper by Giuseppe Crupi, Rosalia Tufano and Gabriele Bavota (Università della Svizzera italiana). The motivation is a company that needs an in-house code generator, for a niche language or its own DSL, and cannot afford the hardware a 30B model needs. Earlier work (RankEF, a CodeT5+ ranker from ASE 2024) showed that sampling several solutions from a small model and letting another small model pick one can help, but never measured how often such a judge calls correct code wrong or wrong code correct. This paper measures that first, then rebuilds the generate-and-select setup with current small models.

Can a small model judge code correctness?

The judges are Qwen2.5 Coder 0.5B and 3B, Gemma-3 4B and Llama-3.2 3B, with RankEF's CodeT5+ 770M and GPT-4.1-mini for comparison. The data is 722 Java tasks from MultiPL-E's HumanEval and MBPP translations and from CoderEval, after dropping tasks whose tests pass on empty or dummy bodies or are flaky. Five small generators wrote 10 implementations per task, and each candidate was labelled by running its tests. The test split holds 5,641 candidates, about 70% of them incorrect. Agreement with the tests is reported as Cohen's kappa.

Zero-shot, small models are poor judges. The best, Qwen2.5 Coder 3B, reaches kappa 0.35 with 36% false positives and 22% false negatives; Llama reaches 0.20 and Gemma 0.10, and the two smallest models behave as constant classifiers. GPT-4.1-mini reaches 0.54, with 23% false positives and 16% false negatives. Two examples in the prompt changed nothing.

Fine-tuning on the labelled training candidates moves every small model into moderate agreement: 0.45 for Qwen 0.5B, 0.57 for Qwen 3B (above GPT-4.1-mini), 0.46 for Llama and 0.49 for Gemma. RankEF reaches 0.40. False positives drop sharply (Llama from 48% to 8%, Gemma from 78% to 15%) while false negatives rise. Training with execution feedback as an extra output target brought no significant benefit, and even the 0.5B Qwen beat RankEF without it. No model, large or small, got past kappa 0.60. In a manual look at 223 of the 528 false positives from the best judge, the misses were mostly algorithmic errors (60%), missing null checks (11%) and wrong assignments (6%).

Confidence is informative. At a 0.9 threshold Llama 3B's precision exceeds 80% on "correct" and is about 90% on "incorrect", at the price of abstaining: 7% of candidates for Qwen, 25% for Llama and 21% for Gemma fall below the threshold. The authors note this suits a reviewer that flags committed code only when sure.

Does a judge make a small generator competitive?

Each small generator (DeepSeek Coder 1.3B, OpenCoder 1.5B, Qwen2.5 Coder 3B, Phi-4 mini, Gemma-3 4B) writes 2, 5 or 10 candidates and one to three fine-tuned judges pick one by normalized confidence. The baseline is the largest model of the same family, run once, with both sides repeated 10 times per task.

For Qwen2.5 Coder 3B, one candidate scores pass@1 0.361 against 0.487 for its largest sibling. With a judge choosing among 2, 5 and 10 candidates it scores 0.443, 0.495 and 0.521. Choosing by the generator's own log likelihood gives 0.397, random choice 0.377, and RankEF 0.452. Across the five generators, 10 candidates and one judge add 12.1, 11.1, 16.0, 20.6 and 5.3 points over a single attempt. Against the large siblings, the small team is significantly better for OpenCoder, Qwen and Phi, statistically tied for DeepSeek Coder despite the 25x size gap, and significantly worse for Gemma, where Gemma-3 4B keeps producing the same candidates and the 27B model was trained on 14T tokens against 4T. One judge was usually enough, and adding more, or combining them with a random forest or a small network, did not help.

The hardware argument is concrete: two RTX 3060 cards (about $600) run a generator and a judge, while a 30B model needs an 80 GB A100 (about $17,500). The team is slower per request at 10 candidates (7.9 s for DeepSeek Coder 33B against 10.4 s for the 1.3B team) but parallelizes on cheap cards.

What kind of verification this is

This is selection among many samples, not review of one change followed by a fix, and the judge is not stronger than the generator. It is specialised: fine-tuned on outputs from the same small generators with test results as labels, so it is judging in-distribution code. That qualifies the common claim that a weak checker is useless. Used zero-shot, these models were close to useless as judges; trained for the job, a 3B model matched a commercial model. Java only, with benchmark tests of variable strength, and the authors ask for replication in other languages.

Cross-references

• cross-model-code-review — how this result qualifies the "weak reviewer is idle" argument
• verification-dynamics-llms — why checking a weak generator's output is easier than checking a strong one's
• rl-finetune-beats-frontier — another case of a small model trained for one narrow judgement beating frontier models
• local-ai-is-not-opus — the deployment economics of small local models