LLM output variance

LLM and agent output varies enough — between runs of the same prompt, and between tasks — that most claims people make from a handful of tries are noise. Dan Luu's treatment in danluu-ai-coding-testing is the sharpest statement of it: whatever conclusion you want about a model or a workflow trick, a small number of runs can support it.

The two axes of variance

• Run-to-run: the same model at the same effort level on the same task scatters. On one of Luu's benchmarks, one standard deviation between runs is ~7.5% performance. The gap between the best and worst models he tested was smaller than one SD of a single model's runs. So "I tried the new model and it's better" is, at n=1–2, indistinguishable from luck.
• Task-to-task: a model that wins on one task loses on the next. With just three hand-made benchmarks, Luu found support for every contradictory claim people made about GPT-5.5 on release ("5.4 is better," "5.5 is cheaper because it just works," etc.) — because each was true on some task.

"Show me the distribution"

The practical upshot for benchmarks: a single summary number ranking models is close to meaningless. Public benchmarks reduce a big diverse task set to one figure, but most tasks are saturated (4/4 for good models, or 0/4 for everyone), and a small subset of decisive tasks determines the ranking. Swap a few of those and the "clearly better" SOTA model flips — Opus 4.8 moves ahead of GPT-5.5, then GLM-5.2 pulls ahead. The high-precision presentation ("1% better at 19% lower cost") implies a resolution the data doesn't have. Luu's analogy: Miguel Indurain is an all-time great or a forgotten time-trialist depending on how long the Tour de France's time-trial stages happen to be — tweak the benchmark, change the legend.

Even benchmarks that seem to measure exactly what you care about can invert in practice. Two benchmarks rank Opus far better than GPT at "not making things up," yet Luu and several others consistently observe Opus 4.x rationalizing nonsense more than GPT-5.x on real debugging — so much that GPT needed less prodding to build a strong game AI, the opposite of the small benchmark's verdict.

The caveman-mode case study

The worked example. "Caveman mode" (a brevity prompt claiming 75% token reduction) was widely recommended; every online "eval" was LLM-generated SEO spam full of errors, and the tool's own author called it a joke. Luu generated three real benchmarks in ~15 seconds each and ran them 50× across models and effort levels. After two runs it looked like a clear win; after 50 the effect averaged to negligible and inconsistent across tasks. The moral isn't just about caveman mode — it's that almost nobody runs enough trials to distinguish a real effect from variance, so recommendations propagate on two-run flukes and superstition. (Matt Mullenweg's framing, via Luu: high-variance activities breed superstition, like gamblers with lucky socks.)

Consequences

• Small-sample model comparisons are unreliable. "X is better than Y" is generally wrong unless the capability gap is large; you can find, without cherry-picking, cases where last-gen Anthropic beats current-gen OpenAI and vice versa.
• Reducing false positives via repetition works for a reason visible in the variance: independent runs / independent agent perspectives cut error rates, which is why agent-failure-mode-skill leans on re-checking and personas.
• Benchmarks serve labs, not users. Public leaderboards don't tell an individual which model to use for their task mix, and revenue trajectories suggest they don't drive real adoption either.

Related: testing-heavy-no-review-workflow (variance is why you can't trust a single test run), reviewing-ai-code, structured-output-benchmark.