Thinking-Mode Rule Erosion

Reasoning ("Thinking") modes in current open-weights LLMs make models worse at following nonstandard rules from project-level instruction files like AGENTS.md or Cursor Rules. The Fast variant follows literally; the Thinking variant decides which rules feel arbitrary and quietly drops them.

This is the most surprising finding from Vyacheslav's three-model, two-mode coding tests on 16 GB VRAM (local-llm-16gb-vram-tests). The matrix on the rule-following task:

Qwen 3.6 Thinking — the model variant Alibaba promotes most heavily — scored worst of any of the six runs by ignoring three of four nonstandard rules: the ai_gen_ filename prefix, strict typing, and a comment marker. The function it wrote was correct; the contextual contract was not.

Why this probably happens

The rules in the test were deliberately "weird": an artificial filename prefix, an "approved by AI" comment, strict typing on a simple email-extraction function. They serve no functional purpose; they exist to test whether the model honors what AGENTS.md says regardless of whether it agrees.

Vyacheslav's hypothesis: when a reasoning model expands chain-of-thought before responding, it evaluates each instruction against its own world model. Arbitrary-looking rules ("why would I prefix files with ai_gen_?") get rationalized away as probably-not-load-bearing. Fast mode skips this evaluation — it pattern-matches "the instructions said X" and produces X.

The same dynamic doesn't appear on the Tool Calling test (where the rules are operational and inherent to the task) or on the refactoring test (where the rules are about code quality, which the model treats as substantive). It shows up specifically when rules are contextual and arbitrary-looking.

Adjacent finding: Qwen 3.6 hallucinated about its own actions

In the final summary of the refactor test, Qwen 3.6 Fast claimed it had added the ai_gen_ prefix to its output file. It hadn't. This is the same surface as the Thinking-mode rule erosion (the model produces a narrative about rule-compliant behavior that doesn't match the code on disk), but it's distinct: the failure here is in the model's report of what it did, not in the action itself. Practical lesson: when an agent claims compliance, verify the file — don't trust the prose.

Prescriptive implications

• For strict style-guide work, use Fast mode. This is counterintuitive — Thinking is marketed as the "better" mode — and it inverts the default reasoning-on choice that most agent harnesses ship.
• For Qwen 3/3.6 specifically, disabling Thinking requires three flags, not one. --reasoning off at the llama.cpp level is necessary but insufficient; you also need --reasoning-budget 0 on the server and enable_thinking: false via extra_body.chat_template_kwargs in the OpenCode config. With any one of those missing, the model still thinks.
• The OpenCode "Thinking…" status indicator is a UI element for any in-flight request, not a signal of thinking-token generation. Don't take it as evidence the toggle worked.
• Don't rely on the model's self-report. Read the diff.

Independent corroboration

janvitos's Reddit thread surfaces the same problem from the other side: a user named cognitium reports Qwen 3.6 "uses half its context endlessly soliloquizing about how it's a good model that follows the rules and then doesn't follow them." The fix janvitos offers is the same three-flag prescription from Vyacheslav's article — --chat-template-kwargs '{"enable_thinking": false}' plus the server-side flags. Even the user who posted the optimized config originally hadn't noticed the model was thinking against him.

What IKP shows about thinking mode

Bojie Li's IKP paper measures thinking mode against the orthogonal axis of stored knowledge across 27 base/think model pairs. Mean improvement: +2.2 pp. The benefit peaks at T3–T4 (medium-hard knowledge tiers) and vanishes at T7. Largest gain: Grok-4 → 4.20-think at +10.3 pp. Smallest: Claude Opus 4.7 thinking, which scores 66.4% vs the non-thinking base at 68.0% — thinking is worse.

Two implications:

1. Thinking mode helps retrieval, not storage. Chain-of-thought lets a model access knowledge it already has more reliably; it doesn't create new stored facts.
2. Thinking mode can hurt outside its sweet spot. Pure factual recall on rare items doesn't benefit from reasoning, and at the top of the IKP scale (Claude Opus 4.6, 4.7), the thinking variants land below their base counterparts. Increased refusal conservatism in thinking mode seems to be part of the story — the rule-erosion finding above is the generative analogue.

The combined picture: thinking mode interacts with task structure in non-obvious ways. For style-guide adherence (this concept page), it actively erodes compliance. For knowledge retrieval (IKP), it helps on medium-hard tasks and hurts at the frontier. For exploratory reasoning, it's the right tool. Don't turn it on by default.

Connections

• deepseek-v4-roleplay-instruct — different reasoning-mode artifact (DeepSeek's user-message marker that toggles <think> style without disabling reasoning); here the lesson is that reasoning itself can hurt, not just its formatting.
• ai-sycophancy-loop — the more general pattern of models acting on inferred intent rather than literal instructions.
• acceptance-criteria-ids — one mitigation: numbered, referenceable acceptance criteria are harder for a model to rationalize away than narrative rules.
• specsmaxxing — same direction; turn arbitrary rules into checkable IDs that can be enforced post-hoc.