#benchmarks
Wiki 19
- A $500 RL Fine-Tune That Beat the Frontier A $500 GRPO fine-tune of a 9B open model beat every frontier config on catalog review, 68× cheaper
- An Empirical Study on Strong-Weak Model Collaboration for Repo-level Code Generation CMU compares strong-weak model splits on SWE-bench Lite; the strong model writing first wins, and no strategy tested is a review stage
- Are Open Models Catching Up? SemiAnalysis reruns era-specific benchmarks on open and closed LLMs and finds catch-up time roughly halving each era, from 19.7 to 4.8 months
- Constraint Decay LLM coding agents look strong on loosely specified tasks but degrade sharply as production structural constraints accumulate
- Constraint Decay: LLM Agents in Backend Code Generation A benchmark showing coding agents lose ~30 assertion-pass points as production structural constraints accumulate
- DeepSeek V4 Flash 0731 scores 50 on the Artificial Analysis Intelligence Index, 10 points above previous DeepSeek V4 Flash Artificial Analysis puts DeepSeek V4 Flash 0731 at 50, one point under GPT-5.6 Luna for ~60% less per task and on the cost Pareto frontier
- Factual Capacity Scaling Stored factual knowledge scales log-linearly with parameters and does not compress — the Shannon-entropy floor that survives the Densing Law
- How far behind are open models? Håvard Tveit Ihle measures open-model lag on 17 benchmarks, finding 8-10 months on private ones, 4-6 on public, and a gap growing since R1
- Improving Code Generation via Small Language Model-as-a-judge Fine-tuned sub-5B judges pick correct Java code about as well as GPT-4.1-mini; small generator plus judge rivals its 8-33B sibling in 4 of 5 families
- Incompressible Knowledge Probes (IKP) Bojie Li's IKP benchmark estimates closed-model parameter counts from factual capacity, R²=0.917 calibration; falsifies the Densing Law for stored knowledge; puts GPT-5.5 at ~9.7T effective params
- little-coder — Scaffold-Model Fit on Aider Polyglot Itay Inbarr shows Qwen3.5-9B Q4_K_M goes from 19.11% (default Aider) to 45.56% (little-coder) on Aider Polyglot by redesigning the scaffold around small-model behavior
- LLMs: Intelligence vs. Cost Guido Imperiale replots Artificial Analysis's intelligence-vs-cost chart on a linear axis with OpenRouter and local-electricity prices
- Open models in perpetual catch-up Nathan Lambert on why the roughly six-month gap between open and closed models holds steady, plus trends in adoption, specialization and China
- Open-closed model gap How far the best open-weight LLMs trail the best closed ones - how it is measured, estimates from 4 to 10 months, and whether it closes
- Scaffold-Model Fit Coding-agent benchmark scores measure model × scaffold, not model alone — small local models inherit responsibilities the scaffold assumed they'd self-manage
- SWE-1.7 — Cognition's Coding Model Cognition's SWE-1.7, RL-trained from Kimi K2.7, reportedly near GPT-5.5 on coding benchmarks — with the strongest numbers on its own FrontierCode
- SWR-Bench: LLM Code Review Comment Generation on Real Pull Requests 1,000 real PRs with full repo context; the best LLM review setups reach about 20% F1, dragged down by false positives, and pooling several reviews helps
- Variation in Verification: Understanding Verification Dynamics in LLMs Salesforce study finding errors from stronger generators are harder for any verifier to catch, so a weak generator plus GPT-4o nearly matches a strong one
- Which Tools Do Claude Code, Codex and Cursor Choose? Armature's 16,893-session study of which third-party services coding agents pick, run by a firm that sells vendors a better pick rate
Toolbox 1
- Agent Reading Test Benchmark that scores how well coding agents read and extract facts from web documentation