Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity

Author: Bojie Li (Pine AI) arXiv: 2604.24827 (April 27, 2026) URL: https://arxiv.org/pdf/2604.24827 PDF copy: ikp-incompressible-knowledge-probes.pdf Code/data: https://github.com/19PINE-AI/ikp Companion site: https://01.me/research/ikp

Summary as captured

Closed-source frontier labs don't disclose parameter counts. The standard alternative is inference economics (estimate size from API throughput and pricing), which carries 2×+ uncertainty from hardware, batching, and serving-stack assumptions. Li's instrument is intrinsic: storing F facts requires at least F/(bits per parameter) weights, so measuring how much a model knows lower-bounds how many parameters it has.

The probe set is 1,400 factual questions across seven obscurity tiers (T1 Universal → T7 Extreme), with tier boundaries calibrated against six landmark open-weight models (Qwen 2.5 0.5B / 7B → Qwen 3 32B / 235B → Kimi K2.5 → Gemini 3.1 Pro). Two probe families: Wikidata founding-year/capital-style attributes (557 probes), and DBLP/arXiv researcher probes (345 probes — "name a paper, named system, institution, or co-author" for a real CS researcher; T5–T7 are dominated by researchers with 10–50 citations).

Scoring: per-probe ∈ {+1.0, +0.5, 0, λ} where λ=−1.0 (penalizes confident bluffing). Tiers can score negative; aggregate is the unweighted mean of the seven per-tier scores.

Calibration. Fit A = α·log₁₀(N) + β on 89 open-weight models (135M–1,600B span). R² = 0.917. Leave-one-out CV: median fold error 1.59×, 68.5% of models within 2×, 87.6% within 3×.

The MoE total-params-predict-knowledge result

For MoE models, total parameters predict knowledge capacity (R² = 0.79) far better than active parameters (R² = 0.51). Factual knowledge is stored across all expert weights, not just those activated per token. Consistent with unified MoE scaling laws (Clark et al. 2022, Ludziejewski et al. 2025, Zhao et al. 2025).

Frontier estimates (out-of-sample, by inverting the calibration)

GPT-5.5 sits 1.4× above the next model and ~2.4× above the GPT-5 / Opus 4.7 / o1 / Grok-4 cluster between 3.0 and 4.1T. The cluster itself: half a decade of competitive frontier development at OpenAI, Anthropic, xAI, OpenAI o-series converging to within 1.4× effective capacity.

Densing Law falsification

Across 96 dated open-weight models (2023-09-27 to 2026-04-24), Li fits:

pen_acc = β₀ + β₁·log₁₀(N_B) + β₂·months

The time coefficient is β₂ = −0.0010/month (95% CI [−0.0031, +0.0008]) — indistinguishable from zero. The Densing Law prediction of +0.0117/month (capability density doubling every ~3.5 months) is rejected at p < 10⁻¹⁵.

The reframing: procedural capability (reasoning, parsing, instruction following) compresses under the Densing Law. Factual capacity does not — it's bounded by Shannon entropy. Both live in the same parameter budget. Benchmark saturation is evidence that benchmarks have stopped measuring the part of scaling that cannot compress, not that scaling has stopped.

Thinking-mode analysis (27 base/think pairs)

Benefit peaks at T3–T4 (medium-hard tiers) and vanishes at T7. Interpretation: chain-of-thought helps with knowledge retrieval, not stored knowledge — consistent with the parameter-count-as-storage framing.

Pro tier finding

OpenAI's "Pro" variants (GPT-5 Pro, GPT-5.2 Pro, GPT-5.4 Pro, GPT-5.5 Pro) gain only +0.3, +0.9, +4.8, +1.0 pp over their non-Pro siblings on IKP. Inverting: the GPT-5 Pro premium is 1.05× and the GPT-5.5 Pro premium is 1.13× in effective factual capacity. Consistent with vendor messaging — Pro tier targets reasoning, agentic, and long-context capabilities (compressible), none of which adds new stored facts.

GPT-5 vs GPT-5.5

GPT-5.x point releases (5, 5.1, 5.2, 5.3, 5.4) cluster tightly around 66.1% IKP, spanning only ~8 pp. GPT-5.5 jumps to 71.4%, well clear of the 5.x cluster. Fingerprint analysis (Section 6.9) shows every consecutive GPT-5 → 5.x transition lands in the retrained regime (Hamming similarity < 0.10, Jaccard collapse), not the lineage regime — point releases are full retrains on different data rather than continued pretraining of a single base. Claude Opus 4 → 4.7 shows monotonic cross-generation increase (52.4 → 65.7 pp) suggesting genuine scale-ups along a single axis.

T6 and T7 boundary

• T6 (Long-tail). Where the proprietary fleet separates from the open-weight curve. DeepSeek V4 Pro: 1.0% on T6. GPT-5.5: 38.5%. Gemini 3.x family has artificially high T6 numbers (53.8–88.5%) because Gemini 3.1 Pro is the T6 calibration landmark — so it's excluded from the frontier table.
• T7 (Extreme). Only Jamba-large (2.8%) and Grok-4 (1.0%) score above 1%. Every other model lands at 0% to within sampling noise. Interpretation: T7 probes target effective mention frequency below what current pretraining corpora preserve. The long tail extends beyond the largest existing model.

Knowledge fingerprinting

Combining rare-fact Jaccard overlap with hallucination similarity (rate at which two models produce the same wrong answer on rare facts) yields a training-free test that distinguishes weight-sharing siblings, post-training lineages, and full retrains across closed-vendor version families — without model weights.

Hubs: arXiv cs.LG.