little-coder — Scaffold-Model Fit on Aider Polyglot

Itay Inbarr's paper holds the model constant and varies the agent scaffold. Same weights (Qwen3.5-9B Q4_K_M, ~6.6 GB), same Aider Polyglot benchmark (225 exercises across Python, Go, Rust, JavaScript, C++, Java), same 32K context, same two-attempt budget. One scaffold is Aider under its own defaults. The other is little-coder, an agent whose design has been adapted specifically to the behavioral profile of a small local model.

Aider with Qwen3.5-9B scores 19.11% (43 / 225). little-coder with the same model weights scores 45.56% mean across two complete runs (SD = 0.94). 79.1% of exercises produced the same pass-or-fail outcome across the two runs — reproducibility well inside the expected noise floor. little-coder outperforms several public-leaderboard entries built on models more than 10× larger, including gpt-4.5-preview and gpt-oss-120b (high). See scaffold-model-fit for the standalone concept.

The scaffold mechanisms

little-coder inherits the multi-turn agent loop and tool abstractions from SafeRL-Lab/clawspring. The adaptations layered on top are the paper's contribution:

Write tool refuses on existing files. When the model invokes Write against a file that already exists, the tool returns a structured error directing it to Edit instead. Without this guard, a whole-file rewrite can silently replace partially-working code. The guard fires on ~57% of exercises — more than half the time, the model would otherwise have wiped working state.

Thinking budget cap with re-injection retry. Reasoning is bounded by 2,048 tokens. When the budget is exceeded, generation aborts, the partial reasoning trace is preserved and re-injected as assistant context, and the request is retried with thinking disabled. This forces the model out of open-ended deliberation into committing to an implementation. The cap fires ~0.90 times per exercise on average — unbounded reasoning would have run past budget on almost every exercise.

Conditional workspace discovery. A knowledge entry fires on coding keywords and directs the model to surface local documentation (.docs/instructions.md, README.md) before editing any code. Glob is invoked on 67% of exercises, Read on 100% — the directive is taking effect.

Tool skill cards instead of a static preamble. Reference material is provided through two small-budget channels rather than a large preamble. Tool skill cards are ~80-150 tokens each, selected per turn by intent prediction, recency, and error-recovery signals. Algorithm cheat sheets are scored against the user message and injected only when their keywords match. Together, the two channels cap at ~500 tokens per turn. This is the explicit answer to Ronacher's complaint about how generic skill systems burn small-model context.

Malformed-output parser + repetition aborter. A wrapper around the agent loop repairs tool calls the model emits in fenced or otherwise non-native form, catches empty responses and hallucinated tool names, and aborts repetitive loops that would otherwise exhaust the turn budget on the same failing approach.

The thesis Inbarr puts on these mechanisms is that a frontier-model scaffold assumes the model self-regulates on each of these dimensions. A small model doesn't. little-coder turns each of them into infrastructure.

The scaffold signal: the asymmetry cell

On 225 exercises:

• Both pass: 32
• Both fail: 113
• little-coder only: 69
• Aider only: 11

The little-coder-only cell is ~6× larger than the Aider-only cell. This is the scaffold signal: on 69 exercises, the model can produce a passing solution, and Aider's scaffold doesn't reach it. The 11 exercises Aider solves and little-coder fails are an honest cost, but they don't undo the asymmetry.

Aider baseline result is itself a finding

The Aider 19.11% baseline is run with the harness's defaults preserved where they describe the scaffold itself (whole-file edit format, temperature 0, two attempts per exercise). Two settings were changed only to prevent the baseline from losing exercises to infrastructure rather than scaffold limitations: num_ctx was raised to 32,768 to match little-coder's context, and litellm's request timeout was raised from 600 s to 1,800 s to prevent long CPU-offloaded generations from timing out. None of the prompts, coder logic, retry mechanics, or edit-parsing code was touched.

The Aider leaderboard does not currently report results for models below ~10B because the working assumption has been that they are not worth a slot. Inbarr's baseline, on a sub-10B Q4 local model, scores above several larger leaderboard entries. The pretty conclusion that follows: small models have been excluded from coding-agent evaluation prematurely. See factual-capacity-scaling and mixture-of-experts for the parallel observation that benchmarks designed to measure procedural capability stopped measuring stored knowledge — Inbarr's point is the symmetric case at the small-model end.

Per-language breakdown

Python is the widest scaffold gap (+35.3 pp); Rust is the narrowest (+13.3 pp). The languages where Aider hurts a small local model most are the ones where the model is otherwise most capable.

Time economics

little-coder costs more wall time per exercise:

Aider's whole-file two-try structure terminates failures quickly (two LLM calls, two test runs). little-coder's agent loop continues exploring up to its 20-turn budget; on hard failures it tends to exhaust most of the budget. Per-hour throughput still favors little-coder (4.7 vs 2.75 passes / hour), but more than half its wall time is spent on exercises it ultimately fails. That's the price of converting the 69-exercise scaffold-delta cell.

Caveats Inbarr puts on the result

• Transfer to SWE-bench, real pull-request workflows, or other small-model backbones is not established. The result is specific to Aider Polyglot with Qwen3.5-9B.
• The mechanism observables (Write-guard fire rate, thinking-budget fire rate, glob/read usage) are not formal ablations. They show each mechanism was active, not what the pass rate would be with it disabled. Per-mechanism ablations are flagged as the natural next step.
• Multi-hour local inference is itself an experimental variable: Ollama's runner subprocess was observed to die and be replaced mid-benchmark on contaminated runs outside the two clean ones reported here, with measurable post-replacement degradation. Inference-runtime stability becomes part of the experimental story when you sit at a desk for 16 hours per Aider run.

Connections

• scaffold-model-fit — the standalone concept
• lucumr-local-models — the "polish gap" Inbarr's scaffold mechanisms try to close in one specific direction
• local-llm-16gb-vram-tests — Vyacheslav's tests use OpenCode + a similar AGENTS.md-style contract; little-coder is the next step out, building scaffold structure that doesn't assume self-regulation
• thinking-mode-rule-erosion — connected via the thinking-budget cap; both arguments end at "stop letting the model decide when to stop"
• clean-code-coding-agents — code-structure-matters-more-with-agents, but from the scaffold side instead of the codebase side
• itayinbarr-blog — Inbarr's blog entity