Are Open Models Catching Up?

SemiAnalysis (Evan Cloutier, Max Kan, Jordan Nanos and Dylan Patel) published this on 21 August 2026, after a two-month run of releases that included GLM-5.3 and kimi-k3. Their opening claim is that these are the first open models able to do paid work, meaning the coding and agentic tasks behind Anthropic's $65B+ ARR. DeepSeek R1 made headlines in January 2025, but in their telling nobody used it for anything economically valuable. The evidence for the shift is token volume: Fireworks alone serves over 40T tokens a day, twice the OpenAI API's volume at the end of March.

The question they set out to answer is the one frontier-lab investors ask. If open models stay close enough to the closed frontier at a fraction of the price, the model layer turns into a commodity and the labs' margins go with it. SemiAnalysis sells a financial model of Anthropic and OpenAI, and links to it three times in the article, so they have a stake in how that question is answered.

One benchmark set per era

The method rests on a claim about benchmarks: each belongs to the period it was built for. A benchmark is designed to separate the models of its day, labs climb it until it saturates, and attention moves to the next one. A single fixed suite applied to 2023 and 2026 models would either floor out at one end or saturate at the other. So the authors split LLM history into three eras (early scaling, reasoning, agentic), choose four benchmarks per era, and score each era's models only on that era's tests.

Within an era, the best result on each benchmark is set to 100 and every other score is scaled against it; a model's composite is the equal-weight mean of its four normalized scores. Most runs were done by SemiAnalysis on Prime Intellect's evaluation stack (the harness in Prime-RL), with Florian Brand of Prime Intellect helping pick benchmarks and check results. The rest come from Artificial Analysis and Datacurve's DeepSWE leaderboard. Open models were served as they would have been at release, with the vLLM version, hardware and sampling settings of the time, and closed models were run against pinned API versions. The authors say benchmark choice was part taste and part popularity: Humanity's Last Exam has known problems but had no substitute in its era, and DeepSWE stands in for the similarly troubled SWE-bench Pro.

The three eras

In the first era the gap opened wide. Llama-2-70B, the first open model that approached the frontier, scored 39.9 against GPT-3.5 Turbo's 75.7, a 35.8-point deficit. Mixtral-8x7B in December 2023 moved toward GPT-4, but GPT-4 Turbo and GPT-4o kept ahead. Llama-3.1-405B finally passed GPT-3.5 Turbo in July 2024 with a composite of 86, and DeepSeek V3 matched GPT-4o in December 2024 (94.1 against 95.5), while Qwen2.5-72B came close at a sixth of the 405B's parameter count. The authors point out that the frontier hardly moved past GPT-4 in this era, because Turbo and 4o were built to be cheaper and faster rather than smarter.

OpenAI's o1-preview on 12 September 2024 reset both the benchmarks and the gap. Grade-school math gave way to AIME, and trivia to Humanity's Last Exam. This time the open side started much closer: DeepSeek R1 opened the era 12.1 points behind. Llama-4 Maverick then stalled the momentum R1 had built, Gemini 2.5 Pro and o3 pushed the frontier further, and the R1-0528 checkpoint closed the initial gap in May 2025 with a score of 78, 8.5 months after o1's debut. Anthropic is absent from the charts for this era. The authors' explanation is that it was not competing for the reasoning leaderboard but turning Claude into the default coding agent, and that set the terms for the third era, whose benchmarks run in a terminal.

The agentic era is dated from Opus 4.5, which the authors say most practitioners treat as the start because of its reliability, even though GPT-5.2 scored higher on their suite. That mismatch is the article's first sign that the full product, model plus harness, had become the thing that mattered. Frontier releases also sped up: OpenAI and Anthropic shipped a flagship every 51 days on average in this era, against 213 days in the first. Kimi K2.6 passed Opus 4.5 in April 2026 with a composite of 56.3, 4.8 months after it launched, and GLM-5.2 cleared GPT-5.2 six months after that model's release, scoring 72.4. At the chart's right edge GPT-5.6 leads at 99.1, followed by Kimi K3 at 93.0 and Fable 5 at 89.5. The two American open models on the chart sit far lower: Nemotron 3 Ultra at 32.7 and Inkling at 51.1.

Reading the halving

The headline is that the time for open models to overtake an era's founding closed model roughly halves each era: 19.7, 8.5, then 4.8 months. The ratios are 0.43 and 0.56, so "halves" is a fair rounding, but three things about the chart limit how far it can be pushed.

Each era contributes one number, so the "remarkably consistent" trend rests on three points. The clocks also start at the closed flagship's public debut, and some of the shrinkage comes from open models entering each era sooner. In the first era the first serious open contender, Llama-2, arrived eight months after ChatGPT. In the third, GLM-4.7 shipped a month after Opus 4.5. And the measure only asks when an open model passes the era's first closed model. It does not ask how far the open frontier sits behind the closed frontier that has kept moving in the meantime. By August 2026 that remaining distance, GPT-5.6 against Kimi K3, is about six composite points.

That distinction matters when this article is set against how-far-behind-are-open-models. Håvard Tveit Ihle, measuring every threshold crossing on 17 benchmarks, finds the gap smallest around R1 in January 2025 and growing since then, especially on private benchmarks. SemiAnalysis's benchmarks are all public and chosen to be recent. Their 4.8 months sits inside Ihle's 4–6 month band for public benchmarks, so the two agree on the rough size of the public gap and disagree on its direction. open-closed-model-gap sets the estimates side by side.

The authors' own caveats

The first is that benchmarks are not the job. Kimi K3 beats Fable 5 on the composite, yet SemiAnalysis still uses Fable for its own daily work. Part of that is productization, since Anthropic ships Claude Code and Claude Tag around the model. The other part is that public benchmarks are easy to climb: a lab can build RL environments that closely mimic the benchmark tasks. That is the same objection local-ai-is-not-opus makes about SWE-Bench, and the one Nathan Lambert makes about the Artificial Analysis index in open-models-in-perpetual-catch-up.

The second is the objection that the third era's closing time is flattered by safety testing, since Anthropic and OpenAI hold models back longer before release than Moonshot and Zhipu do. The authors answer that this is not new. GPT-4 finished training 218 days before it was released, and even if Mythos finished training in mid-February, only 114 days passed before the Fable release. On their numbers the pre-release delay has shrunk, so it cannot explain a faster close. Lambert argues the opposite in glm-5-3-how-chinese-labs-keep-stride: Z.ai releases within days, and the months American labs spend testing are months Chinese labs spend climbing benchmarks.

The article also treats llm-distillation as part of why no lead lasts: "Nothing stays secret forever—especially when you factor in distillation. It's just a question of how long it takes."

What the clip does not contain

The introduction promises to extend the analysis into the future and explain why the result is less bearish for frontier labs than it looks. The clipped text ends after the safety-testing caveat, before either appears, most likely at a paywall. This page cannot report the forecast, and the article's conclusion about frontier-lab margins is missing with it.