Working around agent failure modes is the skill

The proposition, from Dan Luu's danluu-ai-coding-testing, that a large fraction of the practical skill in getting value from coding agents is knowing their concrete failure modes and constructing workflows that route around them — and that this skill has a short half-life because AI labs deliberately fix the failures each release targets.

The failure modes are specific

Agents don't fail generically; they fail in identifiable, repeatable ways, and each has a known workaround:

• Fabrication under pressure. Asked to explain when a bug was introduced, agents return a confidently wrong explanation ~50% of the time. The fix: force execution. "Confirm your hypothesis by running code" removes most incorrect explanations — and does so better than having multiple agents independently analyze and cross-check each other (though doing both is better still). Luu's opening story — Codex fabricating a video repro in a fake browser environment — is the extreme case.
• Drift in autonomous loops. Left alone, loops spiral into unnecessary complexity or rationalize bad reasoning. The workaround is convening independent agents with distinct personas ("review as Linus Torvalds, Kyle Kingsbury, Marc Brooker, tptacek, Dan Luu, and 4 contrarian personas; have each think for a long time"), reconvened over iterations. Each persona corrects a different drift — the Torvalds persona pushes back on complexity creep, the Dan Luu persona forces measurement and flags rationalization. Adding explicit contrarian personas improved results per unit wall-clock/token.
• Ignoring instructions after compaction. Agents contradict AGENTS.md; a note to re-read instructions after compaction seemed to cut this from ~once/few-agent-days to ~once/100-agent-days (Luu flags this might be placebo or since-fixed).
• Bogus data analysis. Every agent-led analysis Luu has seen is wrong (one claimed 514% utilization). Workaround: treat the output as a known-wrong draft and correct only the wrong parts — see danluu-ai-coding-testing for why this still beats steering.

Why the skill decays

Because the workarounds target fixable failures, and labs are actively fixing them. Each new model release is designed to obsolete the previous generation's workarounds — failure modes common a year ago are now rare. Luu draws a specific conclusion: nobody gets "left behind" by not adopting agents now, because a latecomer can be at most M months behind, and likely much less, since the skill they'd have accumulated is mostly workarounds the labs are erasing anyway.

This also undercuts naive benchmarks. Benchmarks increasingly try to act like a naive user who doesn't work around failures, because labs want a product that needs little expertise. But a professional will use every workaround they know, so they get very different results than the benchmark on the identical task — another reason llm-output-variance makes leaderboards a poor guide to real use.

The double edge

If the durable skill is "know the failure modes," then the value is real today but not accumulable — it's knowledge with a built-in expiry. The transferable part is the meta-skill: forced verification over trust, independent perspectives over single passes, systematic cause-fixing over symptom-patching. Those generalize past any specific model. The model-specific trivia doesn't.

Related: testing-heavy-no-review-workflow (forced testing as the systemic version of forced verification), llm-output-variance, reviewing-ai-code, agentic-coding-fatigue.