Why Software Factories Fail

Dex Horthy's argument, subtitled "harness engineering is not enough", is that the lights-off software factory cannot work yet, and that no amount of scaffolding fixes it because the gap is in how coding models are trained. He opens by declaring the bias — he runs HumanLayer, which sells into exactly this problem — and the post ends with a pitch. The middle is the part worth keeping.

This is the same repository as the SlopCodeBench run in benchmarking-opus-5-slopcodebench, and the two documents are in direct conversation: the line Horthy sets in a pull-quote here, "THERE ARE NO GOOD BENCHMARKS for a model's ability to maintain codebase quality," is the claim he partially withdraws in the later post after finding one.

The pitch he is arguing against

The lights-off factory, a term from Dan Shapiro that Simon Willison covered when StrongDM published theirs, is the agentic factory with the human code review step deleted. The reasoning chain is short: you are the bottleneck, the models are good enough, code is free, ship more. OpenAI's Ryan Lopopolo wrote up harness engineering and talked about their factory, Symphony; Ramp, Stripe, WorkOS and Brex have all published on agent platforms shipping on the order of 75% of their code.

The step-by-step is mechanical. Building drops from hours-or-days to minutes-or-hours, so review becomes the bottleneck. You speed review up with agentic review and agentic regression testing, and it stays the bottleneck. You route incidents in so the on-call wakes up to a candidate fix, then user feedback, at which point the job is two questions: how much fits in the queue and how fast can you review the output. Deleting review leaves one question, and the investment moves to sandboxes, orchestration, automated review, monitoring, rollout and feedback signals.

Against that, the Faros AI report on teams that picked up these tools around January: review comments up 25%, comments 22.7% longer, 31.3% of PRs merged with no review at all, incidents per PR up 242.7%, monthly incidents up 57.9%, bugs per developer up 54%. Horthy flags it himself as a correlation signal rather than a smoking gun, in a post whose thesis is to distrust slop data.

HumanLayer ran the experiment in July 2025 and it failed the way he says it always does. You hit one issue the agent cannot solve with any amount of research or reproduction attempts, and you go back into the codebase you stopped reading three months ago while the site is down. The third time, in November, they decided a rewrite was cheaper and his cofounder spent two weeks in VS Code replumbing the patterns by hand.

Why the models do this

The claim is narrow: models cannot maintain and improve codebase quality over time without human steering. Maintainability here means the specific thing where changing one part breaks another — Fowler's shotgun surgery. He grants the models have improved a lot at one-off problems and a marketing site, and says he sees close to no improvement on quality over time, while admitting he cannot prove either direction because nothing measures it. The window before an agent-built codebase starts fighting back has compressed from the ten-year Java sense of brownfield to roughly three to six months.

To explain why, he goes to how the models are trained. The accepted story for why Claude Code took off is not distribution: aider, cline and codebuff all predated it with the same read/write/edit/grep/bash toolset and genuinely good context handling, and they flailed at edits often enough that you would open your editor and do it yourself. Anthropic did RL on the model inside the harness, the first time a lab trained against the exact tools it shipped. The 2024 SWE-Agent paper had already shown how much small tool-shape choices matter — line numbers in read results, find/replace versus line-range edits — and owning the weights turns that from prompt archaeology into training. This is scaffold-model-fit read from the vendor side: if the score is model times scaffold, the lab that controls both terms wins.

The RL loop itself is three steps repeated for weeks: generate agent traces on a task, score them with a verifier, push weights toward the good traces. The scoring is where it goes wrong. Take SWE-bench Multilingual, whose tasks are about fifteen minutes of work scraped from repos like Redis, jq and Django. The reward is one or zero on two conditions: FAIL_TO_PASS, did you fix the reported thing, and PASS_TO_PASS, did you avoid breaking anything else. His worked example is fastlane__fastlane-19304, where a zip action calls .empty? on two optional params and dies with undefined method 'empty?' for nil:NilClass; the human fix defaults both to [] in two lines. The model starts from the commit before the fix, sees the bug report, and never sees the golden patch or the grading test. Its own edits to test files are thrown away — models have been caught commenting out the failing test or splicing in a mock — the benchmark's test patch goes on top, and the suite runs.

Nothing in that scoring can see design. How the model got there does not matter, so nothing penalizes wrapping everything in try/catch or casting away the type system to make the bar go green. Horthy is careful that benchmarks are not verifiers and have to be held out from training; he means the shape of the judgement rather than the specific dataset.

The deeper problem is timing. Tests answer in seconds, which is what makes millions of RL iterations possible, whereas the cost of bad architecture shows up in weeks or months, the first time someone opens the file for a one-line change and finds the change has to happen in eleven places. There is no way to backpropagate an incident to the decision that caused it. And most benchmarks disclose the whole problem up front, which removes any reason to optimize for "easy to change later" — the design gap SlopCodeBench's staged checkpoints exist to attack.

He credits three efforts pointed the right way: SWE-Marathon's roughly 400-hour tasks with a compound reward channel instead of one bit; DeepSWE's tasks built on OSS repos that were never actually built, which fixes contamination but not quality; and Cognition's Frontier Code, which scores multi-PR tasks and does two deterministic things — it penalizes tests that do not fail against the pre-patch code, a mutation-testing move, and runs a judge model over the diff against code-quality rules. His objection to the judge is the sharpest sentence in the post: if a model could reliably tell good code from bad, it would probably have written the good version. RL needs a fast, reliable oracle and maintainability has none. More review agents and more tokens raise the floor by catching dumb mistakes; they do not move the ceiling, because the ceiling is whatever RL managed to teach.

Turning the lights back on

The recovery is not new, and he says so: front-load alignment the way teams did before AI, when planning an hour up front turned a six-hour review into twenty minutes. Four phases, with human judgement in each.

Product review pins down what and why in the user's terms, plus what you will read after shipping to decide it was worth building — a workflow time, an onboarding milestone, an error rate, sometimes just "the support tickets about X stop". He mocks the screens up in rough HTML instead of describing them, on the grounds that a mockup settles an argument three paragraphs would prolong.

System architecture covers how services, endpoints, schemas, queues and stores talk, with sequence diagrams, contract shapes and table definitions. He warns that mermaid can lure you into feeling aligned when you are not, and that architecture alone does not produce good code.

Program design is the phase he calls criminally underemphasized: before anyone writes implementation, agree on the shape of the code. Their first attempt at this was exhausting to read; what worked is light pseudocode. Call-stack trees in diff syntax for control-flow changes, file-tree diffs so you stay in touch with where things live, and the types and method signatures for the key new functions. Each of these is a decision you would otherwise make implicitly during code review, at the most expensive possible moment to change your mind.

Vertical slices counter the horizontal plan models default to — migrations, then services, then API, then frontend — which leaves nothing you can touch until the end. His pre-AI habit was to start in the middle and work outward: mock API contract tested with curl, frontend against the mock, wire to services, then migrations, then business logic and error handling, checking at each step. Before agents, nobody wrote 500 lines without verifying something along the way.

Not everything gets this. Roughly 40% of tasks are one-shot or one-shot with light feedback, medium tasks get product and system design in one document without phasing, and only large work gets all four. He sends off one to three slices at a time and reads the code as they land, because resteering 200 lines is much cheaper than arriving at 2,000 with no idea what broke.

The framing he lands on for PR volume is that you do not have too many pull requests, you have too many bad ones. A PR needing even 20% rework is a burden on submitter and reviewer both, and he puts AI one-shot PRs closer to 50%. That is the throughput ceiling code-review-throughput-limits quantifies and the effort-signal collapse code-review-principal-agent and agent-principal-agent-problem describe, arriving from the training side instead. It sits directly against testing-heavy-no-review-workflow, which answers the same bottleneck by trusting randomized testing instead of readers, and it half-agrees with control-the-ideas-not-the-code: both put the human on design rather than on lines, but antirez is willing to stop reading the diff where Horthy is not. reviewing-ai-code and short-leash-ai-method are the two ends of the same disagreement, and constraint-decay-backend-agents is the closest thing to a measurement of his central claim, showing agents shedding about 30 assertion-pass points as production constraints accumulate.

His closing is a theory of constraints rather than a prediction: learn where the models are weak, build the process around those constraints, seek leverage, read the code. The trade he offers is 2-3x safely against a 10-100x that he thinks is being paid for with codebase quality. Mario Zechner, whose pi-coding-agent is the smallest working example of the opposite instinct, gets cited here for asking the field to slow down, and clean-code-coding-agents arrives at the same discipline from the token-economics direction rather than the maintainability one.