Reviewing AI Code

Thomas Depierre's (thomas-depierre-blog) argument against LLM coding assistants, built entirely on the empirical code-review literature. He's an AI skeptic, but explicitly not for the usual reasons (IP, environment, "it's all crap") — his objection is that he can't see, given the evidence, how these tools help write code better or faster, and that proponents never engage the evidence. Written about a year before publication (he keeps the then-current term "Coding Assistants").

The intern rebuttal and why it fails

The standard defense of LLM coding tools: they're like an intern — expect them to get things wrong, so you review everything, exactly as you already review all code before it enters the codebase. Depierre grants the premise and follows it to its consequence using two well-established empirical findings about code review:

1. A review past ~1 hour hits diminishing returns, regardless of how much code is left. Not because the code is exhausted — because sustained review attention degrades after an hour; the reviewer gets tired and needs time off. He notes there's no research on the recovery interval between sessions, but a realistic ceiling is ~2 effective 1-hour sessions per day, maybe a handful at the extreme.
2. Effective review tops out around 400 LOC/hour. Speed varies with context and experience, but above ~400 LOC/h almost no review reliably finds defects.

Multiply it out: every 400 LOC of LLM output needs roughly one senior developer-hour, and a developer has maybe 10–40 such hours a week — and those high-concentration slots compete with meetings, design, incidents. Best case, a developer writes-reviews-commits a few thousand LOC/day; realistically under 1k LOC/day, and that includes boilerplate, tests, migrations, config. A single unit-test file can exceed 400 LOC. So "just review it" imposes a hard ceiling on how much LLM-assisted throughput is even possible — the review is the bottleneck, not the generation. This is code-review-throughput-limits.

It gets worse: reviewing AI code may be harder

Two compounding problems on top of the ceiling:

• The evidence is human-reviewing-human. All the review research measures humans finding humans' defects. There's no basis to assume humans review LLM output as effectively — and the early, limited evidence points the other way: reviewers of LLM-generated code find fewer defects while being more confident they found them all. The human+LLM pair produces lower-quality output than human+human, but the reviewer believes they did better. So "just review everything" may not even work as a correctness fix, quite apart from the throughput cost. This is the empirical form of the trust problem in credibility-as-slop-test and the practitioner account in know-thine-enemy.
• The flagship use case is the worst-reviewable code. He quotes a prominent "bash the skeptics" post touting "100% of all the Bash code you should author ever again" as something to hand off. Shell is the code that is easiest to get subtly wrong, hardest to review, and most catastrophic on a punctuation typo — exactly the code you least want written by the thing that gets things wrong at random and is (he notes) trained to evade detection. Offering the hardest-to-review code as the best application is where the argument lost him.

He's careful to bound the claim: this is only about the cost of reviewing to flag defects, not the cost of fixing them, and not about how good the model is. Even a very good model hits the same review ceiling.

What would change his mind

He states his falsification conditions, which is the point of the piece:

• Empirical research on humans finding defects in LLM-generated code — how well, how fast, how much per day — reproduced at scale and across contexts. (Current data leans toward humans being bad at it, consistent with models being trained to avoid detection.)
• A demonstration that reviewing LLM output is qualitatively different from reviewing human code, so the existing evidence doesn't transfer. (Current tentative evidence says it is different — but harder, which would strengthen his argument, not weaken it.)

Where it sits

This is one of the sharpest entries in the engineering/craft slot of the anti-llm-discourse map because it's falsifiable and evidence-anchored rather than rhetorical. It's the quantified backbone under several adjacent pages: agent-principal-agent-problem (Crawshaw on the collapse of the effort signal in review), code-review-knowledge-transfer (the empirical finding that review's value isn't mainly bug-catching), stop-using-pull-requests (Laforgia's evidence-based review critique), and no-silver-bullet-llms (Bennett's DORA-scale instability data).

It also stands in direct tension with control-the-ideas-not-the-code: antirez reaches the opposite prescription — stop reviewing LLM code line by line — from the same starting observation that line-by-line review doesn't scale. Depierre's reply is implicit: if review is the only proposed answer to LLM-code correctness and review doesn't scale, then dropping it doesn't solve the problem, it abandons it. The two agree line-by-line review is untenable and disagree on whether anything replaces it.

Dan Luu supplies the third position (testing-heavy-no-review-workflow, from danluu-ai-coding-testing): something replaces review — heavy randomized testing / fuzzing — and it's what should carry reliability instead. He agrees review tops out around 400 LOC/hour and can't gate agent output, but where Depierre reads that as a ceiling on LLM coding, Luu reads it as evidence review was never the right tool. The catch Luu concedes is the precondition Depierre would press: this only works with a testing methodology good enough to trust, which most orgs never built.

Author's frustration

The closing register is anger, not analysis: what drives him is being called "nuts" by people who never engage the question. He's watched the same anecdote-beats-evidence pattern play out in the field before — TDD, type systems, separating test and coding teams, CI/CD, DevOps — and asks that the discussion move to ergonomics and empirical evidence instead of "it worked for me." "I would like if our field stopped at bloodletting."