#llm-evaluation

7 pages search “llm-evaluation” ↳ all tags

Wiki 7

Cross-model code review Who should check an LLM's code, and why the model that wrote it reviews poorly, why the reviewer must be able to solve the task, and when a weaker writer pays
Improving Code Generation via Small Language Model-as-a-judge Fine-tuned sub-5B judges pick correct Java code about as well as GPT-4.1-mini; small generator plus judge rivals its 8-33B sibling in 4 of 5 families
On the Self-Verification Limitations of LLMs on Reasoning and Planning Tasks Stechly, Valmeekam and Kambhampati find GPT-4 self-critique loops lose accuracy while a sound external verifier gains it, even with no critique text
Pine AI Research org behind the Incompressible Knowledge Probes paper; Bojie Li's group, 01.me/research
Reviewer Capability Governs Rejection Targeting, Not Repair Skill A 100-problem pilot where a cross-family reviewer adds 12 points and self-review, despite the best recall, falsely rejects 35% of correct answers
SWR-Bench: LLM Code Review Comment Generation on Real Pull Requests 1,000 real PRs with full repo context; the best LLM review setups reach about 20% F1, dragged down by false positives, and pooling several reviews helps
Variation in Verification: Understanding Verification Dynamics in LLMs Salesforce study finding errors from stronger generators are harder for any verifier to catch, so a weak generator plus GPT-4o nearly matches a strong one