Incompressible Knowledge Probes (IKP)

Bojie Li (Pine AI, arXiv 2604.24827, April 27 2026) builds an instrument that estimates the parameter count of a closed-source LLM by measuring how much it knows. The reasoning is direct: transformer feed-forward layers store roughly 2–4 bits of factual association per parameter (Allen-Zhu & Li 2025; Morris et al. 2025), so storing F facts requires at least F divided by bits-per-parameter weights. Measure F, get a lower bound on parameter count. The conventional alternative — inference economics, estimating size from API throughput and pricing — carries 2×+ uncertainty from hardware, batching, and serving-stack assumptions. IKP's estimator is intrinsic: it uses only black-box API access.

The probe set

1,400 questions across seven obscurity tiers (T1 Universal → T7 Extreme), with boundaries calibrated empirically against six landmark open-weight models — Qwen 2.5 0.5B / 7B → Qwen 3 32B / 235B → Kimi K2.5 → Gemini 3.1 Pro. A probe lands in tier k if the T_k landmark answers correctly and T_{k−1} does not. Non-monotone probes (about 15%) are dropped.

Two probe families dominate. Wikidata supplies 557 founding-year and capital-style attribute probes (T3–T7, sampled by Wikipedia view-count quartile). DBLP/arXiv supplies 345 researcher probes — "name the primary subfield of this CS researcher, and name one paper, system, institution, or co-author associated with their work." T5–T7 are dominated by researchers with 10–50 citations. The remaining 97 probes are manual T1–T4 balance.

Scoring is +1.0 / +0.5 / 0 / λ where λ = −1.0 penalizes confident bluffing. STRONG > WEAK > REFUSAL > WRONG by construction means a calibrated honest model strictly beats an aggressive bluffer at the same knowledge level. Per-tier scores can go negative; aggregate is the unweighted mean of seven tiers.

Calibration

Fit A = α·log₁₀(N) + β on 89 open-weight models spanning 135M to 1,600B. R² = 0.917. Each 10× increase in parameters adds ~14.7 pp in IKP accuracy. Leave-one-out cross-validation: median fold error 1.59×, 68.5% within 2×, 87.6% within 3×.

Three findings worth their own pages

1. MoE total params predict knowledge, not active

For Mixture-of-Experts models, regressing IKP against total parameters gives R² = 0.79. Against active parameters: R² = 0.51. Factual knowledge is stored across all expert weights, not only those activated per token. This validates a piece of folk knowledge that didn't have a clean experimental anchor before; see mixture-of-experts which has been updated with this result. Consistent with Ludziejewski et al. (2025) joint MoE scaling laws and Clark et al. (2022) unified MoE scaling.

2. The Densing Law is false for stored knowledge

The "Densing Law" (Huang et al. 2025) claims capability per parameter doubles every ~3.5 months. Across 96 dated open-weight models from 2023-09-27 to 2026-04-24, IKP fits:

pen_acc = β₀ + β₁·log₁₀(N_B) + β₂·months

β₂ = −0.0010/month (95% CI [−0.0031, +0.0008]) — indistinguishable from zero. The Densing prediction of +0.0117/month is rejected at p < 10⁻¹⁵.

The reframing matters: procedural capability (reasoning, parsing, instruction following) is a compressible function over its inputs, and the Densing Law is real for that. Factual knowledge is incompressible — bounded below by Shannon entropy of the stored facts. Both share a parameter budget. Benchmark saturation is therefore evidence that standard benchmarks have stopped measuring the part of scaling that can't compress, not evidence that scaling has stopped. See factual-capacity-scaling.

3. The frontier estimate table

By inverting the calibration on 92 proprietary models (none used to fit the curve), Li produces out-of-sample effective-size estimates:

Gemini 3.x family is excluded from the frontier table because Gemini 3.1 Pro is the T6 calibration landmark — its T6 is 88% by construction and inflates the within-family curve.

Four bands resolve out:

• Top: GPT-5.5 alone, 1.4× above the next model, 2.4× above the broad cluster.
• Frontier cluster: GPT-5 / Opus 4.7 / o1 / Grok-4 / o3 / Opus 4.5 / 4.1 between 3.0 and 4.1T. Five years of competitive development at four labs converging within 1.4× effective capacity.
• Older flagships: GPT-4.1, Grok-3, GPT-5.4 Pro, older Opus generations at 2–3T.
• Small/efficient line: GPT-5 Mini 410B, Gemini 2.5 Flash 207B, GPT-5 Nano 71B, Claude Haiku 4.5 65B. Spans an order of magnitude in capacity. The ratio between top and bottom of the proprietary fleet (~150×) is almost exactly the ratio between the largest and smallest open-weight model in the calibration set.

Smaller findings

Pro tier adds little factual capacity. GPT-5 Pro premium is 1.05× and GPT-5.5 Pro premium is 1.13× in effective capacity. Consistent with vendor framing — Pro tiers buy reasoning, agentic, and long-context capabilities (compressible procedural), not new stored facts.

GPT-5 → 5.x point releases are retrains, not lineage. The 5.x cluster (5, 5.1, 5.2, 5.3, 5.4) spans only ~8 pp around 66% IKP. GPT-5.5 jumps to 71.4%, well clear of the cluster. Fingerprint analysis (rare-fact Jaccard and hallucination similarity) puts every 5 → 5.x transition in the retrained regime (similarity < 0.10), suggesting OpenAI ships point releases as full retrains on different data, not continued pretraining of one base. Claude Opus 4 → 4.1 → 4.5 → 4.6 → 4.7 shows monotonic IKP gains (52.4 → 65.7 pp on the non-thinking base), consistent with single-axis scale-up.

Thinking mode does not create stored knowledge. Across 27 base/think pairs, mean improvement is +2.2 pp; benefit peaks at T3–T4 and vanishes at T7. The largest gain is Grok-4 → 4.20-think at +10.3 pp. Claude Opus 4.7 thinking is worse than non-thinking on IKP (66.4 vs 68.0). Interpretation: chain-of-thought helps retrieval, not stored facts. Updates thinking-mode-rule-erosion with the IKP-specific evidence.

Per-tier discrimination.

• T1–T2 saturate by ~120B.
• T3 (steepest slope, 0.324/decade) saturates at ~1.2T and is the single most informative tier; ρ = 0.983 with overall accuracy.
• T6 is where the proprietary fleet separates from the open-weight ceiling: DeepSeek V4 Pro scores 1.0%, GPT-5.5 scores 38.5%.
• T7 is a hard ceiling. Only Jamba-large (2.8%) and Grok-4 (1.0%) score above 1%. Every other model at 0% within sampling noise. T7 anchors the obscurity ladder beyond the largest existing model — interpreted structurally, T7 probes target effective mention frequency below what current pretraining corpora preserve.

Knowledge fingerprinting

Combining rare-fact Jaccard overlap with hallucination similarity (rate at which two models produce the same wrong answer on rare facts) yields a training-free instrument that distinguishes weight-sharing siblings, post-training lineages, and full retrains across closed-vendor families. The interesting thing here is that wrong answers on rare facts are a stronger fingerprint than right ones: two models that learned the same wrong fact almost certainly share training data.

Connections

• mixture-of-experts — updated with the total-vs-active capacity result
• factual-capacity-scaling — the standalone concept that survives outside this paper
• thinking-mode-rule-erosion — updated with IKP thinking-mode evidence
• pine-ai — Bojie Li's organization
• deepseek — V4 Pro shows up as the open-weight ceiling at T6
• chatgpt-5-5-pro-mathematical-research — Gowers et al on GPT-5.5 Pro; IKP independently confirms 5.5 as the genuine scale-up jump from the 5.x point-release cluster