Twelve Ways to Be Wrong About AI-Assisted Coding

Greg Wilson's May 2026 post (greg-wilson-blog) starts from a mundane situation: your manager wants proof that the AI coding subscription pays for itself. You could count lines of code, count closed tickets, or send a survey. Each of those is wrong, and each is wrong in a different, nameable way. The post walks through twelve such errors and maps each to a failure that the human sciences catalogued long ago — Goodhart's Law, the Hawthorne effect, selection bias, internal validity. The framing note matters: this is about how people assess AI-assisted coding, not a claim that the coding itself is bad. With light rewording the same twelve apply to claims made for agile, TDD, and most other process fads.

The twelve

1. Counting lines generated. LLMs produce more code, not better outcomes. A 40% rise in lines per developer measures verbosity. Deleting 2,000 tangled lines and writing 200 clean ones is progress that shows up as a loss on this metric, and every extra line is future reading, maintenance, and debugging that the count never charges you for.
2. Timing artificial tasks. The famous "55% faster with Copilot" result (Peng 2023) was implementing an HTTP server in JavaScript from scratch in ninety minutes with no other obligations. Real work is navigating a codebase you didn't write, ambiguous tickets, meetings. Speed on a greenfield toy predicts none of it.
3. Before/after with no control group. PRs ship faster in June than January, but you also hired twelve engineers, rebuilt CI, and switched cloud providers. Without a group that didn't adopt the tools you can't attribute the change to the tools. Internal validity needs a credible counterfactual.
4. Asking whether developers feel more productive. "87% report feeling more productive" (Liang 2024) is contaminated three ways: Hawthorne effect (observed people behave differently), novelty effect (new tools feel fast, fades in weeks), and social-desirability bias (people say what the survey wants, more so when management chose the tool).
5. Counting commits, PRs, and tickets. McKinsey proposed exactly this in 2023. Goodhart's Law: track commit count and you get more, smaller commits; track tickets and tickets get split. The numbers move, the work doesn't.
6. Measuring only the easy half. Generation is easy to measure; review, debugging confidently-wrong suggestions, security holes, and technical debt are not. One analysis of 300,000+ AI-authored commits found over 15% introduced at least one quality issue, and nearly a quarter of those persisted long-term (Liu 2026). A 2025 evaluation of five major LLMs found none produced web-app code meeting industry security standards.
7. Adoption rate as success. "90% adoption across engineering" is a procurement outcome — the tool is installed and opened. It says nothing about whether suggestions are useful or correct.
8. Volunteers vs non-volunteers. Comparing developers who chose the tool against those who didn't compares two populations, not two conditions. Early adopters are already more motivated and often higher-performing — selection bias. It's the most common design flaw in industry reports because it's the cheapest study to run; one longitudinal study found Copilot users were more active than non-users before the tool arrived (Stray 2026).
9. Individual instead of system. If code gets written 30% faster but ticket-to-production time is flat, writing code was never the bottleneck. One study found AI helped junior contributors but senior developers lost 19% of their own productivity absorbing a 6.5% rise in review load from AI-generated code (Xu 2025). Optimizing one pipeline stage while ignoring the rest is a systems-thinking failure.
10. Measuring during the novelty period. A four-week study that finds a boost has found a four-week boost. The effects that matter — skill atrophy, debt accumulation, changed collaboration — surface over months. An analysis of 807 open-source repos adopting Cursor found exactly the split: a large but transient velocity gain alongside a persistent rise in complexity and static-analysis warnings (He 2026).
11. Suggestion-acceptance rate as quality. Acceptance measures whether code looked plausible enough to press Tab, not whether it was correct or secure. An enterprise study of 400 developers found a 33% average acceptance rate with high satisfaction and no tracking of correctness or security at all (Bakal 2025). Deadline pressure raises acceptance for the wrong reasons.
12. Comparing AI to nothing. A control group using nothing is a baseline that doesn't exist in practice. Real developers without an LLM still have documentation, colleagues, and thinking time. The question that matters is whether the tool beats those alternatives, and it's rarely the one asked.

The counter-evidence, gathered

The single strongest data point Wilson cites is the randomized controlled trial (Becker 2025 — the METR study) in which AI tools increased completion time by 19% for experienced open-source developers, who had predicted the opposite for themselves. Put next to the Copilot "55% faster" number, the two studies measure almost nothing in common: one is a scoped greenfield sprint, the other is real maintenance work with a control. The rest of the pattern is consistent — senior-developer productivity down 19% under a 6.5% review-load increase, transient velocity with persistent complexity across 807 repos, over 15% of 300k+ AI commits carrying quality issues, and a 33% acceptance rate that no one paired with a correctness measure.

Why this belongs in the wiki

This is the methods backbone for the whole measurement thread. The concept page measuring-ai-coding-productivity pulls the recurring failures — Goodhart, control groups, novelty, selection, individual-vs-system — into one place. danluu-ai-coding-testing makes the same "show me the distribution" argument from a practitioner's bench: single summary numbers hide run-to-run variance wide enough to support any conclusion. no-silver-bullet-llms supplies the DORA/CircleCI instability data that error #9 predicts. constraint-decay is the mechanism behind error #2 — agents look strong on loose greenfield tasks and degrade as production constraints pile up. reviewing-ai-code is error #9 stated as a throughput cap, and dont-outsource-learning is the long-run skill-atrophy cost that errors #4 and #10 keep failing to measure.