Map

Factual Capacity Scaling

Wiki conceptllmscaling-lawsbenchmarks ↳ show in map Markdown
title
Factual Capacity Scaling
type
concept
summary
Stored factual knowledge scales log-linearly with parameters and does not compress β€” the Shannon-entropy floor that survives the Densing Law
tags
llm, scaling-laws, benchmarks
sources
ikp-incompressible-knowledge-probes
created
2026-05-13
updated
2026-05-13

LLM parameters serve at least two functions: storing facts and computing functions. The two have different scaling behavior. Procedural capability β€” reasoning, parsing, instruction following β€” compresses under newer architectures, training recipes, and data mixes; this is the Densing Law (Huang et al. 2025), capability-per-parameter doubling every ~3.5 months on standard benchmarks. Factual knowledge does not compress. It's bounded from below by the Shannon entropy of the stored associations: storing F facts requires at least F divided by bits-per-parameter weights, and bits-per-parameter has empirical ceilings around 2–4 (Allen-Zhu & Li 2025; Morris et al. 2025).

Bojie Li's IKP benchmark instruments this distinction directly. Across 96 dated open-weight models from late 2023 through April 2026, fitting pen_acc = Ξ²β‚€ + β₁·log₁₀(N_B) + Ξ²β‚‚Β·months produces Ξ²β‚‚ = βˆ’0.0010/month, 95% CI [βˆ’0.0031, +0.0008] β€” indistinguishable from zero, and rejecting the Densing prediction of +0.0117/month at p < 10⁻¹⁡. Factual capacity does not compress.

Why this matters

The widely-reported "saturation of reasoning benchmarks" β€” MMLU, HELM, math contests β€” was read by many as evidence scaling has stopped paying off. The IKP measurement says no: saturation is evidence that the benchmarks have stopped measuring the part of scaling that can't compress. Reasoning benchmarks measure procedural capability, where the Densing Law is real. Factual benchmarks designed to probe the long tail of obscure knowledge keep producing log-linear gains with parameter count.

The implication for capability forecasting: if your forecast was "model capability plateaus by 2027 because benchmarks saturate," you were extrapolating from the wrong measurement. Larger models keep storing more facts at a roughly constant log-linear rate, and the long tail of knowledge is bottomless β€” IKP's T7 tier scores 0% on every current model including ones estimated at multi-trillion effective parameters.

The decomposition

N_total = N_fact + N_proc + N_ling

  • N_fact β€” share devoted to entity attributes, dates, names, specific propositions. Lower-bounded by Shannon entropy of the stored facts; incompressible.
  • N_proc β€” share devoted to reasoning, parsing, instruction following, tool use. Compressible; Densing Law applies.
  • N_ling β€” share devoted to syntax, morphology, vocabulary, register. Compressible.

Better architectures and training recipes improve the efficiency of N_proc and N_ling, freeing capacity for N_fact or allowing smaller total N. They cannot reduce N_fact for a fixed corpus of facts. IKP probes N_fact specifically.

The 2–4 bits per parameter range

Allen-Zhu & Li 2025 establish that transformer models store approximately 2 bits per parameter of factual knowledge under ideal training, with empirical estimates ranging up to ~3.6 bits per parameter for GPT-style models (Morris et al. 2025). Lu et al. 2024 show fact capacity scales linearly with model size and exponentially with training epochs, estimating ~1,000B parameters to memorize all Wikidata facts at 100 epochs. The 2–4 bits/param range provides the upper bound on what facts a given parameter budget can store; IKP measures what facts it does store given a real training corpus.

What IKP estimates

Inverting the calibration curve fit to 89 open-weight models lets you estimate the effective parameter count of any closed-source model from its black-box API. Five dollars of API spend on the 1400-probe set produces a parameter estimate with median 1.59Γ— error, 87.6% of estimates within 3Γ— of true size. This is the only intrinsic estimator of closed-model size in the literature β€” alternatives are inference-economics-based (Epoch AI 2024) and carry 2Γ—+ uncertainty from serving-stack assumptions.

See incompressible-knowledge-probes for the full frontier-estimate table; some headline numbers: GPT-5.5 ~9.7T, Claude Opus 4.6 ~5.3T, GPT-5 ~4.1T, Gemini 2.5 Flash ~207B, Claude Haiku 4.5 ~65B.

Three corollaries worth flagging

MoE total params predict knowledge, not active. RΒ² of 0.79 against total parameters vs 0.51 against active. Factual knowledge is stored across all expert weights, not only the ones routed to per token. See mixture-of-experts.

Pro tier adds little factual capacity. GPT-5 Pro premium is 1.05Γ— effective capacity; GPT-5.5 Pro is 1.13Γ—. Pro tiers buy reasoning, agentic, and long-context capabilities (compressible procedural), not new facts.

Thinking mode helps retrieval, not storage. Mean +2.2 pp on IKP across 27 base/think pairs, with the benefit peaking at T3–T4 and vanishing at T7. Stored knowledge is fixed; chain-of-thought changes what gets accessed.

Symmetric observation at the small end

scaffold-model-fit makes the parallel point about coding-agent benchmarks: scores measure model Γ— scaffold interaction, not model alone, and benchmarks designed for one regime mismeasure the other. Both arguments point at the same structural failure mode: benchmarks freeze assumptions about what part of capability they're measuring, and capability shifts can outpace those assumptions in either direction.

Cross-references

Linked from