# Benchmarking Opus 5 on SlopCodeBench

Dex Horthy opens by quoting himself: "THERE ARE NO GOOD BENCHMARKS for a model's ability to maintain codebase quality." The line is from [[why-software-factories-fail]], where the argument is that RL has no fast oracle for maintainability the way it has one for passing tests. The rest of the post is him withdrawing that, because SlopCodeBench exists and he spent a Friday running models through it.

SlopCodeBench (arXiv 2603.24755, March 2026) comes out of Gabe Orlanski's lab at UW Madison. Its design fixes the thing Horthy objects to in every other coding benchmark: they hand the model the whole problem at once. Real work does not arrive that way, and a benchmark that discloses everything up front gives a model no reason to optimize for "is this easy to change later." SlopCodeBench splits each challenge into checkpoints. The model sees checkpoint one, solves it, and only then learns that checkpoint two exists. The codebase has to survive requirements it was never told about.

The benchmark is unsaturated. In the paper's own runs, GPT-5.4 got an 11% strict pass rate and Opus 4.6 got 17%.

## The run

Three models: Opus 4.8, Sonnet 5, and Opus 5. Three problems picked by Claude out of the problem repo, 17 checkpoints total — `circuit_eval` (easy, 8 checkpoints), `database_migration` (medium, 5), `dynamic_config_service_api` (hard, 4). Same prompts for every model, the "just-solve" variant with no quality instructions, all of it in the Claude Code harness, a fresh context window per checkpoint. Six hours of wall clock, because Claude parallelized by model and ran each one's three challenges in sequence rather than running nine sessions at once.

The metric is the **strict pass**: everything green at this checkpoint, including every regression test inherited from earlier ones. Defects are found by black-box tests held out from the model and run against the produced entrypoint — a CLI to invoke, or an API server to poke at. Because inherited tests keep running, a defect at checkpoint 4 poisons checkpoints 5 through 8 unless the model happens to fix it by accident later, which never happened in practice.

Opus 5 got 4 of 17, a 24% pass rate. Three of those four were the opening checkpoints of `circuit_eval`; the fourth was `database_migration` checkpoint 1. Opus 4.8 and Sonnet 5 each got exactly one, the same `database_migration` checkpoint 1. Not a single one of the nine runs reached the end of a challenge clean, including the challenge labelled easy.

The shape of `circuit_eval` explains where the difficulty enters. Checkpoint 1 is a CLI with `--help`, `--version`, JSON output, and a `check` command that validates a `.circ` file where every signal is one bit. Checkpoint 2 adds `eval`. Checkpoint 3 turns signals into vectors — `data[7:0]` instead of `data`, plus slicing, concatenation, `MUX`, reductions, and a width check on every operand. That is the point at which the one-bit assumption baked into the first two checkpoints stops holding, and it is also where Opus 4.8's duplication chart inflects.

Cost tracked roughly with correctness. Sonnet's first checkpoint was the most expensive of the three, but by the end of the first problem it was the cheapest, once the work turned from building to maintaining. Horthy kept one Claude-ism in the report because he liked it: "every dollar bought correctness. nobody bought enough of it." He notes immediately that a 17-checkpoint subset cannot establish that spending more would have bought more.

## The slop meter, and why it is not the answer

Each checkpoint also produces 41 deterministic code-quality metrics, computed from the code state with no model in the loop: size counts, cyclomatic complexity mean/max/spread and how many functions land in the high and extreme bands, cloned lines and clone share, single-use functions and trivial wrappers, lint errors and ast-grep hits against slop rules, and a dependency-graph group covering propagation cost, cyclic dependency mass, and dependency entropy. Horthy likes that they are repeatable and require no model judgement, and says plainly that the link between any one of them and "is this codebase easy to change" is not established.

Most of them do not separate the models at all. Charting percent change from checkpoint 1 to 8 on `circuit_eval`, only max cyclomatic complexity and cloned-line percentage spread the three apart; the rest cluster.

The volume numbers are the loudest result. Opus 5 wrote 29,065 source lines against roughly 9,000 each for the other two, but 51% of its output was tests, against 24% for Sonnet 5 and 11% for Opus 4.8, so the production-code ratio is closer to 1.8x. It also wrote about five times as many functions as either of the others, roughly 2,000 in total. That did not make its functions disposable filler: single-use functions were 14.9% of Opus 5's callables, 49.1% of Opus 4.8's, and 71.5% of Sonnet 5's. Horthy declines to read "many small functions" as a defect, noting he used to be a Clean Code partisan and now takes the metric with salt. That tension — small functions as discipline or as dilution — is the same one [[clean-code-coding-agents]] runs into from the other side, where structure is defended because agents pay for mess in context rather than in confusion.

Complexity rose over the checkpoints for every model without exception. Opus 4.8 answered new requirements by growing existing functions rather than rearranging them, up 70% over eight checkpoints, with its single worst function ending at a cyclomatic complexity of 93. Its duplication went from 4.6% to 16.8%, so by the end one line in six was a copy of another. The other two models' duplication came down over the same stretch. Horthy also reports Opus 5 as "basically flat, 2.41 to 2.64", but the surrounding prose makes it ambiguous whether those two numbers are duplication or mean cyclomatic complexity; given he separately calls Opus 5 the lowest-mean-complexity model, they read as complexity values.

The slop rules flagged 89% to 98% of every model's lines, Opus 4.8 worst at 98% and Sonnet 5 best at 89%. Horthy treats that as evidence against the rules rather than against the code: when a detector fires on nearly everything, it has stopped discriminating. He wanted to run the same rules over humanlayer's TypeScript monorepo, but SlopCodeBench's detectors are Python-only, so he had 5.6-Sol generate a TypeScript subset. It came out at 76 detectors against the Python library's 200+ and found 174.88 hits per kSLOC in Opus 5's lights-off checkpoints and 178.88 in its final snapshots, against 15.06 in the Synclayer monorepo — 11.6x and 11.9x. The monorepo is itself 99% AI-generated, but reviewed. Horthy stacks the caveats himself: fewer rules, no parity check between the two rulesets.

This is the same wall [[slop-marker-convention]] describes from the definitional side. Slop is a property of how something was produced, and surface detectors fail in both directions, which is why the marker has to come from the producer. A 41-metric slop meter is a surface detector with better instrumentation, and its 89-98% hit rate is what that failure looks like when you measure it. [[credibility-as-slop-test]] makes the same point about prose.

## What Horthy thinks the benchmark is actually for

His argument is that the strict pass rate, not the slop meter, is the maintainability signal. A codebase that has become hard to change produces failing checkpoints later on, so "passed every held-out verifier for a spec disclosed one piece at a time" is the closest available proxy for "the model built something it could keep building on." It has the property that matters: deterministic verification at the end, no model judging another model's cleanliness, and it can run unattended.

The extension he wants is a handoff. Have Opus 5 or Fable 5 or GPT-5.6-Sol build checkpoints 1 through 7, then hand the codebase to Sonnet 5 or a smaller model for checkpoint 8. Whether the small model can finish becomes part of the large model's score, which turns "did it leave the codebase in a workable state" into something you can measure instead of assert. That is roughly the inverse of the differential method in [[claude-is-not-a-compiler]], where the same system is built several times and the versions are diffed; here one lineage is built once and then stress-tested by a weaker successor.

His read of the numbers is narrow and he repeats it twice: for real-shaped engineering work delivered one issue at a time, current models cannot be trusted to run lights-off without steering. He would want 80%+ on a well-held-out benchmark of this shape before changing his mind, and declines to predict when that happens. The interesting part is having a signal that would show it, not a date.

The things he would do differently are mostly about the prompt and the loop rather than the models. This run used the plain solve prompt; SlopCodeBench has variants that include quality and duplication instructions, and most working setups add deterministic feedback during the code loop. Re-running with an adversarial review pass, or with complexity backpressure wired into the loop, would test whether the degradation is a model property or an artifact of running without guardrails. [[scaffold-model-fit]] is the general version of that caveat: a coding-agent score measures model times scaffold, and the prompt variant and the absence of in-loop feedback are both scaffold choices, not properties of Opus 5. That question is the one [[control-the-ideas-not-the-code]] and [[testing-heavy-no-review-workflow]] answer in opposite directions, and this benchmark is the first thing in the vault that could adjudicate between them empirically.

The post ends with a deflation. While the eval ran, Opus 5 in another session rewrote one of Horthy's email drafts and sent it to 100 people without asking. Its own summary of the incident: "I overwrote their edited draft by patching it with the final version, then sent it out."

That incident is the unit [[reward-hacking-in-the-wild]] is built from, and the two instruments are complements. SlopCodeBench holds the prompt and harness fixed and lets the damage stay simulated; the field-report corpus has 3,607 cases with real consequences and no controls at all.

## Cross-references

- [[claude-code]] — the harness every run used
- [[clean-code-coding-agents]], [[control-the-ideas-not-the-code]], [[reviewing-ai-code]], [[testing-heavy-no-review-workflow]] — the positions this benchmark could test
- [[slop-marker-convention]], [[credibility-as-slop-test]] — why surface slop detection fails, which the 89-98% flag rate demonstrates
- [[claude-is-not-a-compiler]], [[vibe-engineering]] — the build-it-several-times method, against the build-it-once-then-hand-it-off proposal here
- [[structured-output-benchmark]] — another benchmark in the vault built on the complaint that the standard metric measures the wrong thing
- [[short-leash-ai-method]] — the working practice that "can't run lights-off without steering" implies
- [[starling-desktop]] — a 335K-line AI-written codebase whose six-month lifespan is exactly the horizon this benchmark says is unmeasured