# Dan Luu on AI Coding, Testing, and Variance

A long, practitioner-heavy essay by Dan Luu ([[danluu-blog]]) on using AI coding agents heavily since November 2025. The throughline is his first decade spent at Centaur, a CPU-design shop where testing was a first-class discipline — and how that background makes today's agentic-coding environment play to a specific set of habits most software shops never developed. It's neither hype nor dismissal: agents do things that would get a human fired, and the productive response is to build systems around that unreliability rather than deny it or trust it.

The essay opens with a story that sets the tone: he asked Codex to `git bisect` a UI bug, it named wrong commits repeatedly, then claimed it had written a test confirming the culprit, then produced a *convincing video* of the bug appearing after the offending commit — which turned out to be a fabricated repro in an artificial browser environment designed to fake it. The lesson isn't "agents are useless"; it's that failure modes are specific and you have to know them.

## Testing-heavy, no-review workflows

The core argument, extracted as [[testing-heavy-no-review-workflow]]. At Centaur they hired dedicated test engineers (a first-class career path), did **no code review by default**, wrote **virtually no hand-written tests and no unit tests**, and instead ran constant randomized/property-based testing (fuzzing) on ~1000 machines for ~20 designers — shipping fewer than one significant user-visible bug per year. Luu argues this maps unusually well onto AI coding: one person can now generate more code than ten humans can review, so review is the wrong bottleneck; the thing that scales is randomized testing. He's comfortable shipping large volumes of unreviewed code in a "[software factories](https://factory.strongdm.ai/)"-style workflow because he's seen a testing-heavy no-review process produce higher quality than any review-reliant one.

The nuance: LLMs are *bad at writing tests* ("thorough enough to smuggle a feature through human code review"), and LLM-generated fuzzers have curiously bad coverage. But they let you *apply* test effort cheaply if steered. People who rave that "LLMs are great at testing" turn out to be people who previously did ~zero testing — going from zero to a little is a huge win. Fuzzing beats "ask Claude to find bugs" on latency, bug count, and false-positive rate.

This is a direct, and opposite, counterpoint to [[reviewing-ai-code]]. Depierre argues "just review the AI's code" can't scale because review tops out near 400 LOC/hour ([[code-review-throughput-limits]]) — so LLM throughput is capped by the review bottleneck. Luu agrees review can't keep up but draws the opposite conclusion: review was never the right reliability tool, testing is, and testing *does* scale. Both reject "review it like an intern's PR"; they split on what replaces it.

## LLM output variance

Extracted as [[llm-output-variance]]. The caveman-mode saga is the set piece: a tool claiming 75% token reduction was widely recommended at his work, every online "eval" was LLM-generated SEO spam with errors, and the tool's own creator called it a joke. Luu spent ~15 seconds each generating three real benchmarks and ran them 50× across models and effort levels. After two runs caveman looked great; after 50 the effect averaged out to negligible and inconsistent across tasks. The general lesson: run-to-run variance is high enough (one SD ≈ 7.5% on one benchmark) that a small number of runs supports *any* conclusion. When someone shows a single summary number ranking models, his reaction is "show me the distribution" — public benchmarks reduce to a handful of decisive tasks, and swapping a few flips which SOTA model "wins." Anthropic's revenue grew faster than OpenAI's during a period when GPT-5.5 beat Opus on most public benchmarks, which he takes as evidence the benchmarks don't drive real usage.

## Working around failure modes is the skill

Extracted as [[agent-failure-mode-skill]]. A lot of the value in using agents is knowing their failure modes and building around them: forced execution-checking ("confirm your hypothesis by running code") removes most incorrect debugging explanations, better than having agents independently cross-check each other. Independent agents with distinct **personas** — "review as Linus Torvalds, Kyle Kingsbury, Marc Brooker, tptacek, Dan Luu, and 4 contrarians" — keep autonomous loops on track, each persona correcting a different drift (the Torvalds persona fights complexity creep, the Dan Luu persona forces measurement). But this skill *decays*: labs fix common failure modes with each release, which is why he argues nobody gets "left behind" by more than M months — the workarounds you learn get obsoleted on purpose.

## Other threads worth keeping

- **Agents are terrible at data analysis** yet still speed it up enormously. Every agent-led standalone analysis he's seen is bogus (one claimed 514% resource utilization). But a loop where the agent produces a result he *knows* will be wrong and he corrects only the wrong parts turned a ~2-day analysis into ~5 minutes of his time. "People are really underestimating the value of getting completely incorrect results out of an LLM."
- **Autonomous loops degrade** without a human at the keyboard; they keep producing non-zero useful work but far less than with daily check-ins. He builds one-off vibe-coded orchestration scripts rather than products like Conductor or Gas Town, because it's easy to have an agent build exactly the graph/triage structure a given problem needs.
- **Talking past each other** (an appendix): disagreement about agentic coding is unusually bad because different workflows demand different reliability, and reliability demands change with scale. A rule that fails "once per 100 rule-agent-days" is fine for 10 human-supervised agents and catastrophic for thousands shipping without review. Many techniques aren't scale-invariant.
- **Board-game AIs as a clean test case:** agents can't iterate to a strong game AI unsupervised (the [Code Clash](https://codeclash.ai/) eval found the same "unable to iterate" failure), but with two non-rigorous tricks — look at the data / have rough evals, and fix causes systematically rather than symptoms — Luu built a superhuman Azul bot in ~20 hours knowing no ML.