# How far behind are open models?

Håvard Tveit Ihle, who runs the private WeirdML benchmark, posted this analysis on LessWrong in May 2026. The clip was taken from the greaterwrong mirror and carries no date, so the month comes from Nathan Lambert's [[open-source-ai-reading-list]]. It asks how many months open-weight models trail the best closed ones, and whether that number is shrinking. The data is 17 benchmarks, 8 private and 9 public, giving about 110 datapoints drawn mostly from Epoch AI's Benchmarking Hub. The code and data are on [GitHub](https://github.com/htihle/open_closed_gap).

## Result

As of spring 2026, open models are roughly 8–10 months behind the closed frontier on private benchmarks, whose test data is not publicly available, and roughly 4–6 months behind on public ones. Both trends reached their low point around DeepSeek R1 in January 2025 and have risen since.

![[how-far-behind-are-open-models-2c6eb20b.png]]

The chart is the argument in one picture. The public trend (blue) falls from about 12 months in 2023 to about 3 months in early 2025, then climbs back to about 5 by April 2026. The private trend (red) starts in mid-2024, drops to about 6 months around R1 and rises to nearly 9 months by the time of GLM-5.1. Each point is one benchmark threshold, placed at the date an open model first crossed it, with its height showing how much earlier a closed model had crossed the same threshold.

The public and private sets share no benchmarks, and Ihle takes the fact that they trend the same way as a sign that the trend is real, though he says that is suggestive rather than proof. The other reading is about the level, not the trend: public benchmarks show roughly half the gap private ones do. Public benchmarks still carry information about capability, but they understate the distance by almost a factor of two.

## How the gap is measured

Each benchmark gets a ladder of score thresholds, usually at 5% steps. For each threshold, the gap is the time between the first closed model crossing it and the first open model crossing it. Ihle's example is o1-preview, released 12 September 2024, crossing several thresholds that DeepSeek R1 crossed on 20 January 2025, which gives datapoints of about 4.3 months each, dated January 2025.

The method assumes each benchmark tested every major model, and none of them did. So each threshold was checked by asking whether the recorded first-crossers are plausible, and whether a missing model would probably have crossed first and changed the gap a lot. Datapoints that failed were rejected. Claude Opus 4.7 made those judgements and wrote the justifications, which are in the repository. Ihle then reviewed them by hand and overruled some, in every case to *accept* a datapoint Opus had rejected as too uncertain. Opus also wrote essentially all the code and one of the appendices.

Dating each gap at the open model's release makes it a backward-looking measure. It answers "how far back do I have to go to find closed models as good as today's best open ones?", or in Ihle's words, "how long-lived are the gaps that a top open model closes when it's released?". The forward-looking question, how long until open models match today's closed frontier, is usually the more interesting one, but it can only be answered after the fact. The backward-looking version also avoids a bias: thresholds that closed models have crossed and open models have not are left out, instead of being counted as short gaps because they are still open.

## Biases in both directions

Ihle names biases that push the estimate each way, which is what makes the result usable.

The method has a winner's-curse bias: the first model over a threshold tends to be one that got a lucky run, and since benchmarks usually test more closed models than open ones, closed models get more lottery tickets. That inflates the gap. So does serving. Private benchmarks are often run on open models through third-party providers with zero data retention, so the test data does not leak, and WeirdML, METR and Epoch AI all do this. Third-party hosts sometimes serve open models with subtle bugs that are hard to detect, which would widen the gap, more so on the private side.

Contamination on the private side cuts the same way. Epoch runs FrontierMath, but OpenAI funded it, has access to most of the problems and reported its own o3 scores. ARC-AGI's "semi-private" set is sent to commercial APIs during evaluation, and closed models receive it through their own first-party APIs while open models receive it through third-party hosts. Both effects inflate closed scores, make the closed frontier cross thresholds earlier, and so overstate the gap.

On the other side, the difference between the public and private results points at open-model developers not fully filtering benchmark data out of training, or training toward the tests. Ihle suspects something similar happens with private benchmarks too, to a smaller degree, because labs train on verifiable tasks and verifiable tasks are what benchmarks are made of. Closed labs have more varied data, more enterprise feedback and less need for benchmark scores, so he speculates that the gap on real-world tasks is larger than even the private benchmarks show. He flags this as speculation not drawn from the data. [[local-ai-is-not-opus]] makes the same argument from practice.

## Provenance audit

An appendix, written by Opus and lightly edited, audits who ran each benchmark's evaluations. The clean core, where one independent party ran every model in a fixed harness, is GPQA Diamond, MATH Level 5 and OTIS Mock AIME (all run by Epoch), plus WeirdML, SimpleBench and METR Time Horizons. GSM8K, MMLU and MMLU-Pro are largely vendor-reported or community-submitted, Aider Polyglot and Terminal-Bench mix submitted configurations, and Humanity's Last Exam has an extra problem. Epoch's data had almost no open Chinese models on it, so five scores were added by hand from self-reported sources, and every open-side first-crossing on HLE in the analysis comes from those rows.

## Side analyses

Grouping benchmarks by category shows a larger gap for reasoning. All three reasoning benchmarks are private, though, so Ihle attributes the difference to that and says the data is too thin to say more about categories.

Restricting the open side to Chinese models gives essentially the same curves back to Llama 3.1 in July 2024, with a clearly larger gap before that. Read the other way, the open frontier has been Chinese since mid-2024. That matches the reading list's statement that the leading open models have all come from Chinese labs since about 2024.

## Against the other estimates

This is the most pessimistic of the reading list's estimates about direction. [[are-open-models-catching-up]] reports catch-up time halving each era, and the reading list itself says the gap has shrunk to 4–6 months. Ihle's public-benchmark number agrees with that size. What he adds is the private-benchmark number, nearly twice as large, and a trend that has been widening since January 2025. [[open-models-in-perpetual-catch-up]] had argued in February 2026 that the gap was holding steady at about six months, and that the Artificial Analysis index understates it. The comparison across all of these is on [[open-closed-model-gap]].