The Myth of unsafe Open Source AI

Florian Brand published this on 10 June 2026. He takes the standard case against open models, that they cannot be controlled after release and can be fine-tuned for anything, and accepts it as true in theory. He objects to what it assumes: that closed models are safer, that their guardrails hold, and that providers stop misuse. To test that, he went through third-party threat-intelligence reports on real incidents and listed which models were used. Nathan Lambert's open-source-ai-reading-list summarizes it as showing that closed-model guardrails "are regularly bypassed causing a plethora of real AI-risks before hypothetical risks of open weight models have emerged".

Method

Brand deliberately avoided benchmarks. Attackers break guardrails in ways benchmarks do not measure, and many cyber benchmarks reward novel exploits when "the real world runs on outdated software". He used reports from security firms without privileged access to any provider's usage data, because a lab's own threat report can only see its own platform. He states the obvious limit himself: it is hard to know when and how open models are used, since local inference leaves no provider trail. He also says the search was done with ChatGPT and Gemini at their highest settings, and that he wrote the report himself.

The incidents

The cyber list is dominated by agentic closed tools. In the Mexican government hack, from December 2025 to February 2026, a single operator used claude-code and GPT-4.1 over the API to exfiltrate 195 million taxpayer records. Claude's guardrails were bypassed with an AGENTS.md file, and the models did most of the exploit chaining with occasional nudges, during and after Anthropic introduced Constitutional Classifiers++. The Bissa Scanner mass-exploited known bugs such as React2Shell using Claude Code and OpenClaw on Claude Sonnet 4.6, and stole records, API keys and files. An individual compromised hundreds of badly configured FortiGate gateways with Claude and deepseek plus a set of scripts. PromptSpy is Android malware that uses Gemini as a computer-use agent to trap users in a malicious app. It needed sideloading and had no known real-world impact, but shows where computer-use agents lead. Patriot Bait was a Russian-speaking actor who used Gemini CLI, jailbroken through a GEMINI.md file, to pose as an American veteran, spread misinformation, run pump-and-dump schemes, hack WordPress sites and steal credentials. North Korean actors used ChatGPT's image generation to forge South Korean ID cards for spearphishing.

Open models appear, but less often. The EvilTokens phishing-as-a-service used GPT-4o-mini to translate emails and Llama 3.1 8B and Llama 3.3 70B to analyse them and pick targets open to social engineering. Llama's guardrails were bypassed with a prompt. In LameHug, which Ukraine's CERT found in mid-2025, APT28 used Qwen2.5-Coder-32B-Instruct to generate malicious commands at runtime on victims' machines. The model is open-weight, but the attackers called it through the Hugging Face inference API. So even this open-model case ran through a hosted endpoint that could in principle have been watched.

On misinformation, the Pravda network seeds pro-Kremlin and pro-Iranian content that ChatGPT, Claude, Gemini, Grok and DeepSeek then pick up as sources. Brand includes it despite his no-benchmarks rule because NewsGuard reports it is working better over time. CopyCop (Storm-1516) runs hundreds of fake news sites and likely used an uncensored Llama 3.1 8B. Grok's image generation produced sexual images, including of minors, and Nazi and ISIS propaganda, and a February 2026 Reuters report found it still did so after the guardrails were tightened.

The exception is child sexual abuse material and pornography. There, fine-tuned open image and video models and LoRA adapters are the norm, and offenders advise each other to avoid closed models because they log requests. Brand calls the Internet Watch Foundation's report "pretty damning", and the IWF itself says it sees only part of the problem.

The argument

Brand concludes that closed models are the main vehicle for malicious use, except for deepfakes and CSAM. Every major lab's guardrails, frontier models included, fall to a single prompt or a config file. His explanation is incentives: "Why secretly acquire hundreds of GPUs, collect a ton of data, and hire researchers with the expertise needed to fine-tune a less capable model for a single use case when you can just add a CLAUDE.md file to a frontier model and use it to break into governments?" Attackers use closed models because those are at the frontier or easier to set up.

He also questions the one real advantage of closed models, revocability. No model has been withdrawn. The trend is to ship stronger models to more customers faster, and once misuse has happened it cannot be undone. A provider can only tighten classifiers or quietly weaken the model afterwards.

He expects the response that this looks backwards, and calls it "a convenient argument because it is impossible to refute with evidence". The same warning came with ChatGPT, GPT-4, o1, Sonnet 3.6 and Opus 4.5. Each time open models caught up at a fraction of the price, and the misuse frontier stayed closed. He sees "no indication that Mythos or Fable will be any different."

Where it collides with other pieces

That last sentence is where a-safe-path-to-open-weights disagrees most directly. Thinking Machines, a month later, cites Mythos Preview's autonomous exploit-writing as a reason indiscriminate release is not safe. Brand's evidence cannot settle that dispute. It covers models up to early 2026, and the Thinking Machines argument is about capability that had not yet diffused to open weights. The two pieces also differ on attacker sophistication: Brand describes the lazy attacker who picks the easiest frontier tool, while the staged-release case is built around capability a defender-only window can protect.

The survey is the empirical fourth step of the marginal-risk framework in societal-impact-of-open-foundation-models, which warned against assuming closed releases are safe by default. Its CSAM finding matches the one vector where that 2024 paper also found strong evidence of open-model risk. national-ai-cybersecurity-policy makes a related point from the defender side: harm depends on attackers and defenders, not on one model's benchmark score. anthropic-threat-report-september-2026 is an example of the lab-side report Brand chose not to rely on.

Two limits go unaddressed. Absence from third-party reports is weak evidence for local open-model use, which is by nature what investigators are least likely to see. Brand concedes the difficulty but still reads the tally as a measure of prevalence. And the list counts incidents rather than weighting them, so a sideloaded proof-of-concept sits next to a 195-million-record breach. The case against closed-model guardrails is strong. The case that open models are safe rests mostly on how rarely they appear.