# Scaffold-Model Fit

Coding-agent benchmark scores measure the interaction between a model and the scaffold that wraps it, not the model alone. The standard evaluation practice treats the scaffold as a constant and the model as the variable — under that framing, a 9B local model's Aider score "is" the model's score. But the scaffold encodes assumptions: how much autonomy to expect, how aggressively to edit files, how to recover from failures, whether reasoning is bounded. Those assumptions were made for frontier-class models. When you swap in a model that can't self-regulate the same way, the failures get blamed on model weights when the scaffold is the binding constraint.

Itay Inbarr's [[little-coder-scaffold-model-fit|little-coder paper]] is the cleanest demonstration in the wiki. Same model (Qwen3.5-9B Q4_K_M), same benchmark (Aider Polyglot), same hardware. Scaffold A is Aider's default. Scaffold B is little-coder, redesigned around the model's behavioral profile. Scaffold A scores 19.11%, scaffold B scores 45.56%. **The 26.5 pp gap is the scaffold signal.**

## What scaffolds assume that small models can't deliver

Five dimensions where frontier-model scaffolds expect self-regulation:

1. **File-write discipline.** A frontier model knows when to `Edit` instead of `Write` an entire file. A 9B model will whole-file-rewrite a partially working implementation roughly half the time. little-coder's `Write` tool refuses to operate on existing files and returns a structured redirect to `Edit` — fires on ~57% of exercises.

2. **Bounded reasoning.** Frontier models terminate chain-of-thought when they have enough to act on. Small models will deliberate past any reasonable budget. little-coder caps reasoning at 2,048 tokens, re-injects the partial trace as context, and retries with thinking *disabled*. Fires on ~0.90 exercises per benchmark on average.

3. **Workspace discovery.** Frontier models surface project documentation organically. Small models will plow into editing without reading `README.md`. little-coder injects a conditional knowledge entry that fires on coding keywords and directs the model to glob and read first. `Glob` fires on 67% of exercises, `Read` on 100%.

4. **Reference material packaging.** Frontier models tolerate a large static preamble. Small-model context budgets won't survive it. little-coder uses two small-budget channels: tool skill cards (~80-150 tokens, selected per-turn by intent prediction), and algorithm cheat sheets (keyword-matched). Combined cap: ~500 tokens / turn.

5. **Recovery from output malformation.** Frontier models emit clean tool calls. Small models will fence them in code blocks, hallucinate tool names, and loop on the same failing approach. little-coder wraps the agent loop with a malformed-output parser, an empty-response detector, and a repetition aborter.

Each of these is a place where a frontier-model scaffold *assumed self-regulation* and a small-model scaffold *turns the assumption into infrastructure*.

## The asymmetry cell as the scaffold-signal estimator

A clean way to measure scaffold-model fit: cross-tab the two scaffolds' exercise-level outcomes.

|  | Aider pass | Aider fail |
| --- | --- | --- |
| little-coder pass | 32 | **69** |
| little-coder fail | 11 | 113 |

The 69 cell is exercises the *same model weights* can solve, but Aider's scaffold doesn't reach. The 11 cell is the honest cost of switching scaffolds. The asymmetry between 69 and 11 is the scaffold-fit signal. The ratio (~6×) gives an order-of-magnitude estimate of how much scaffold redesign can recover for a fixed model.

## Why the leaderboard exclusion is a category error

Aider's public Polyglot leaderboard does not report results for models below ~10B parameters. The implicit assumption: such models aren't worth a slot. Inbarr's *baseline* run (default Aider scaffold, Qwen3.5-9B) scores above several leaderboard entries built on much larger models. The exclusion isn't measuring what it thinks it's measuring — it's encoding the scaffold assumption that frontier-model behavior is the only behavior worth benchmarking.

This parallels the [[incompressible-knowledge-probes|IKP paper]]'s argument at the other end of the size axis: benchmarks designed to measure procedural capability have stopped measuring stored knowledge, so the Densing Law looks true on benchmarks while being false on factual capacity. Both arguments point at the same structural problem — benchmarks freeze their assumptions, and capability shifts can outpace those assumptions in either direction.

## Practical implications

- **Comparing models on a shared scaffold is informative only when the scaffold fits both.** Two models with very different self-regulation profiles need different scaffolds to be evaluated fairly. The leaderboard view ("rank models on this benchmark") implicitly assumes the scaffold doesn't change the ranking.
- **Mechanism observables are not ablations.** little-coder's mechanism-fire-rates show that each piece was active during the benchmark, not that any one of them is responsible for a specific portion of the gain. Designing a scaffold around a model's behavioral profile is an integrated piece of work; you can't necessarily decompose it into independent contributions.
- **"This local model is bad at coding" usually contains a scaffold claim.** Try a scaffold that doesn't assume frontier-model self-regulation before concluding the weights are the binding constraint.

## Cross-references

- [[little-coder-scaffold-model-fit]] — the source paper
- [[lucumr-local-models]] — Armin Ronacher's complementary argument that the *toolchain* polish is the gap; scaffold-model fit is the layer above
- [[clean-code-coding-agents]] — code-structure-matters-with-agents; this is the symmetric observation at the scaffold side
- [[local-llm-16gb-vram-tests]] — Vyacheslav uses [[toolbox/opencode|OpenCode]] with `AGENTS.md` rules; OpenCode is *less* frontier-model-shaped than Aider but still less small-model-shaped than little-coder
- [[thinking-mode-rule-erosion]] — the thinking-budget cap in little-coder is one mechanism for this; "stop letting the model decide when to stop" is the shared move
- [[acceptance-criteria-ids]] — turn arbitrary rules into IDs that can be verified, which is what a thinking-budget cap with retry effectively does
- [[incompressible-knowledge-probes]] — symmetric argument at the frontier-model end about benchmark blind spots
- [[swe-1-7]] — a model trained inside and shipped only in its own scaffold (Devin), so model and scaffold can't be factored apart — the extreme end of scaffold-model coupling