LLMs: Intelligence vs. Cost

Artificial Analysis publishes an Intelligence Index, the mean score across a fixed set of benchmarks it runs on every model, and records what each run cost it. Because the benchmark set is the same for every model, that cost works as a relative price per task. Its best-known chart plots one against the other and draws the Pareto frontier, the cheapest model at each intelligence score. Guido Imperiale, a staff software engineer at OpenTeams and a Dask maintainer, argues that the chart misleads and redraws it llm-intelligence-vs-cost.

The publisher has a stake in the conclusion. OpenTeams' page title carries its tagline, "AI you own", and the post ends by recommending cheap open-weights models and local inference. The post contains no pitch, but that is where its conclusion leads.

Three complaints

The first is the logarithmic cost axis. A log scale is what lets a reader separate a model costing $0.015 per task from one costing $0.032 on the same chart as one costing $3.69, about 250 times more. The price is that the reader stops seeing how large the gap between cheap and expensive models is, and how little the differences among the cheap ones matter.

The second is that Artificial Analysis prices every model at its developer's own API. For open-weights models that can cost far more than renting the same weights from a third-party host, and OpenRouter makes switching to the cheapest host easy.

The third is that small local models appear at datacenter prices, which are always poor value for the intelligence bought, and which nobody running such a model on their own machine pays.

The redraw

His plots keep Artificial Analysis's intelligence scores, at maximum thinking effort unless stated otherwise, and change the rest. The cost axis is linear, "because people's money is not logarithmic". Kimi K3, Qwen3.8 Max, DeepSeek V4 Flash 0731, GLM-5.3, GLM-5.3-Flash and Hy3 are priced at OpenRouter rates, excluding providers that are very slow, unreliable, or lack zero-data-retention policies. GLM-5.3-Flash at high effort is extrapolated by combining Artificial Analysis's max-effort score with Z.ai's own coding scores at other effort levels. Models under 35 billion parameters are priced as local electricity. Ornith-1.5-35B-A3B is added from its authors' self-reported benchmarks, which he says deserve "a healthy dose of skepticism". The original Artificial Analysis Pareto chart is reproduced on deepseek-v4-flash-0731-artificial-analysis, and z-ai covers the lab behind GLM-5.3.

He offers a reading rule for the intelligence axis: a one-point difference is unlikely to be noticed, a five-point gap is substantial. A score of 50, the floor of his high-intelligence plot, is roughly what the best model in the world delivered in February 2026, Opus 4.6. The charts themselves are images and are not in the clipped text, so individual points other than those quoted below cannot be checked from the source.

Pricing a local model

The electricity estimate is crude and says so. Take output tokens per task from Artificial Analysis, observe decode speed on local hardware (mostly an RTX 3090 from 2020, about $1,400 used), measure the difference between peak and idle power draw, price electricity at $0.2049/kWh (the population-weighted US residential average as of May 2026), and add 15% for uncached input tokens and waiting on tool calls. Hardware counts as free, on the argument that an RTX 3090 PC or a 64 GB Strix Halo is a machine people want anyway. Platform barely matters: a Strix Halo draws less power than a 3090 but runs longer.

That argument stops working above 64 GB of memory, which almost nobody buys except for AI. Qwen3.8-Flash needs at least a 128 GB Strix Halo, which costs $3,600 against $2,000 for the 64 GB model and enables nothing except models in the ~120B class. His table of what larger local models need: 128 GB (Strix Halo at $3,600, DGX Spark at $4,300, Mac Studio M5 Max at $5,100, MacBook Pro M5 Max at $7,150) runs Qwen3.8-Flash, and GLM-5.3-Flash and DeepSeek-V4-Flash with degraded intelligence; 256 GB (two DGX Sparks at $8,700, Mac Studio M5 Ultra at $11,300) runs GLM-5.3-Flash and DeepSeek-V4-Flash properly; 512 GB (two M5 Ultras, $22,600) runs GLM-5.3; and Kimi K3 needs 2 TB, two TensTorrent Galaxy Blackhole systems at $320,000.

The conclusion

State-of-the-art models from Anthropic and OpenAI are, in his words, too expensive even for large corporations, while the cheaper Chinese models can cost about as much as a phone subscription. Returns diminish fast. Fable 5.1 scores 66 at $3.69 per task. GLM-5.3 scores 60 at $0.49, 7.5 times cheaper, and he thinks most people would struggle to tell the difference. GLM-5.3-Flash at high effort scores 55 at $0.023, 160 times cheaper than Fable; it is visibly weaker at sophisticated work such as one-shotting a whole coding project, but he calls it enough for 90% of what people need. Qwen3.8-27B scores 52 for $0.015 of electricity on a computer a gamer already owns.

What the argument does and does not show

The log-scale point is about perception rather than information: a 250× ratio is visible on a log axis as distance, and a linear axis turns it into a dollar gap instead. Which one is right depends on whether the question is "how much will this cost me" or "how many times more". The pricing corrections carry more weight, because they move models on and off the frontier rather than just redrawing it.

What the plots cannot carry is the "90% of what people need" claim, which is a judgement with no measurement behind it. Per-task cost on a benchmark suite also leaves out what a wrong answer costs in retries and review, which is the point of local-ai-is-not-opus: a local Qwen handles bounded tasks well and falls short on long-horizon ones, and that gap is invisible in a single averaged index. The one-point noise rule sits alongside llm-output-variance, and several of the points that matter most for his conclusion are extrapolated or self-reported.

The dollar figures matter for anyone paying per token, which is the enterprise problem ai-token-budget-explosion describes. OpenRouter-style host switching is the llm-api-routing-layer pattern. The larger local configurations connect to moe-cpu-offload, kimi-k3 and deltafin, and qwen3-8-flash-next covers a Qwen3.8-Flash model of the ~120B class that his hardware table places at 128 GB.