# Incompressible Knowledge Probes (IKP)

Bojie Li (Pine AI, arXiv 2604.24827, April 27 2026) builds an instrument that estimates the parameter count of a closed-source LLM by measuring how much it *knows*. The reasoning is direct: transformer feed-forward layers store roughly 2–4 bits of factual association per parameter (Allen-Zhu & Li 2025; Morris et al. 2025), so storing F facts requires at least F divided by bits-per-parameter weights. Measure F, get a lower bound on parameter count. The conventional alternative — inference economics, estimating size from API throughput and pricing — carries 2×+ uncertainty from hardware, batching, and serving-stack assumptions. IKP's estimator is *intrinsic*: it uses only black-box API access.

## The probe set

1,400 questions across seven obscurity tiers (T1 Universal → T7 Extreme), with boundaries calibrated empirically against six landmark open-weight models — Qwen 2.5 0.5B / 7B → Qwen 3 32B / 235B → Kimi K2.5 → Gemini 3.1 Pro. A probe lands in tier *k* if the T_k landmark answers correctly and T_{k−1} does not. Non-monotone probes (about 15%) are dropped.

Two probe families dominate. Wikidata supplies 557 founding-year and capital-style attribute probes (T3–T7, sampled by Wikipedia view-count quartile). DBLP/arXiv supplies 345 researcher probes — "name the primary subfield of this CS researcher, and name one paper, system, institution, or co-author associated with their work." T5–T7 are dominated by researchers with 10–50 citations. The remaining 97 probes are manual T1–T4 balance.

Scoring is +1.0 / +0.5 / 0 / λ where λ = −1.0 penalizes confident bluffing. STRONG > WEAK > REFUSAL > WRONG by construction means a calibrated honest model strictly beats an aggressive bluffer at the same knowledge level. Per-tier scores can go negative; aggregate is the unweighted mean of seven tiers.

## Calibration

Fit `A = α·log₁₀(N) + β` on 89 open-weight models spanning 135M to 1,600B. R² = 0.917. Each 10× increase in parameters adds ~14.7 pp in IKP accuracy. Leave-one-out cross-validation: median fold error 1.59×, 68.5% within 2×, 87.6% within 3×.

## Three findings worth their own pages

### 1. MoE total params predict knowledge, not active

For Mixture-of-Experts models, regressing IKP against **total** parameters gives R² = 0.79. Against **active** parameters: R² = 0.51. Factual knowledge is stored across *all* expert weights, not only those activated per token. This validates a piece of folk knowledge that didn't have a clean experimental anchor before; see [[mixture-of-experts]] which has been updated with this result. Consistent with Ludziejewski et al. (2025) joint MoE scaling laws and Clark et al. (2022) unified MoE scaling.

### 2. The Densing Law is false for stored knowledge

The "Densing Law" (Huang et al. 2025) claims capability per parameter doubles every ~3.5 months. Across 96 dated open-weight models from 2023-09-27 to 2026-04-24, IKP fits:

```
pen_acc = β₀ + β₁·log₁₀(N_B) + β₂·months
```

β₂ = **−0.0010/month** (95% CI [−0.0031, +0.0008]) — indistinguishable from zero. The Densing prediction of +0.0117/month is rejected at p < 10⁻¹⁵.

The reframing matters: procedural capability (reasoning, parsing, instruction following) is a *compressible* function over its inputs, and the Densing Law is real for that. Factual knowledge is *incompressible* — bounded below by Shannon entropy of the stored facts. Both share a parameter budget. Benchmark saturation is therefore evidence that standard benchmarks have stopped measuring the part of scaling that can't compress, not evidence that scaling has stopped. See [[factual-capacity-scaling]].

### 3. The frontier estimate table

By inverting the calibration on 92 proprietary models (none used to fit the curve), Li produces out-of-sample effective-size estimates:

| Model | Vendor | IKP | Est. size | 90% PI |
| --- | --- | --- | --- | --- |
| GPT-5.5 | OpenAI | 71.9% | **~9.7T** | [3.2–28.7T] |
| Claude Opus 4.6 | Anthropic | 68.0% | ~5.3T | [1.8–15.6T] |
| GPT-5 Pro | OpenAI | 66.5% | ~4.1T | [1.4–12.2T] |
| GPT-5 | OpenAI | 66.4% | ~4.1T | [1.4–12.1T] |
| Claude Opus 4.7 | Anthropic | 66.4% | ~4.0T | [1.4–12.0T] |
| o1 | OpenAI | 65.4% | ~3.5T | [1.2–10.3T] |
| Grok-4 | xAI | 64.8% | ~3.2T | [1.1–9.4T] |
| Claude Sonnet 4.6 | Anthropic | 60.9% | ~1.7T | [579B–5.1T] |
| Gemini 2.5 Pro | Google | 58.4% | ~1.2T | [387B–3.4T] |
| GPT-4o | OpenAI | 55.3% | ~720B | [241B–2.1T] |
| Qwen3-Max | Alibaba | 55.0% | ~685B | [229B–2.0T] |
| GPT-4 | OpenAI | 54.8% | ~666B | [223B–2.0T] |
| Gemini 2.5 Flash | Google | 47.4% | ~207B | [69B–617B] |
| Claude Haiku 4.5 | Anthropic | 39.9% | ~65B | [22B–194B] |

Gemini 3.x family is excluded from the frontier table because Gemini 3.1 Pro is the T6 calibration landmark — its T6 is 88% by construction and inflates the within-family curve.

Four bands resolve out:

- **Top:** GPT-5.5 alone, 1.4× above the next model, 2.4× above the broad cluster.
- **Frontier cluster:** GPT-5 / Opus 4.7 / o1 / Grok-4 / o3 / Opus 4.5 / 4.1 between 3.0 and 4.1T. Five years of competitive development at four labs converging within 1.4× effective capacity.
- **Older flagships:** GPT-4.1, Grok-3, GPT-5.4 Pro, older Opus generations at 2–3T.
- **Small/efficient line:** GPT-5 Mini 410B, Gemini 2.5 Flash 207B, GPT-5 Nano 71B, Claude Haiku 4.5 65B. Spans an order of magnitude in capacity. The ratio between top and bottom of the proprietary fleet (~150×) is almost exactly the ratio between the largest and smallest open-weight model in the calibration set.

## Smaller findings

**Pro tier adds little factual capacity.** GPT-5 Pro premium is 1.05× and GPT-5.5 Pro premium is 1.13× in effective capacity. Consistent with vendor framing — Pro tiers buy reasoning, agentic, and long-context capabilities (compressible procedural), not new stored facts.

**GPT-5 → 5.x point releases are retrains, not lineage.** The 5.x cluster (5, 5.1, 5.2, 5.3, 5.4) spans only ~8 pp around 66% IKP. GPT-5.5 jumps to 71.4%, well clear of the cluster. Fingerprint analysis (rare-fact Jaccard and hallucination similarity) puts every 5 → 5.x transition in the *retrained* regime (similarity < 0.10), suggesting OpenAI ships point releases as full retrains on different data, not continued pretraining of one base. Claude Opus 4 → 4.1 → 4.5 → 4.6 → 4.7 shows monotonic IKP gains (52.4 → 65.7 pp on the non-thinking base), consistent with single-axis scale-up.

**Thinking mode does not create stored knowledge.** Across 27 base/think pairs, mean improvement is +2.2 pp; benefit peaks at T3–T4 and vanishes at T7. The largest gain is Grok-4 → 4.20-think at +10.3 pp. Claude Opus 4.7 thinking is *worse* than non-thinking on IKP (66.4 vs 68.0). Interpretation: chain-of-thought helps retrieval, not stored facts. Updates [[thinking-mode-rule-erosion]] with the IKP-specific evidence.

**Per-tier discrimination.**

- T1–T2 saturate by ~120B.
- T3 (steepest slope, 0.324/decade) saturates at ~1.2T and is the single most informative tier; ρ = 0.983 with overall accuracy.
- T6 is where the proprietary fleet separates from the open-weight ceiling: DeepSeek V4 Pro scores 1.0%, GPT-5.5 scores 38.5%.
- T7 is a hard ceiling. Only Jamba-large (2.8%) and Grok-4 (1.0%) score above 1%. Every other model at 0% within sampling noise. T7 anchors the obscurity ladder *beyond* the largest existing model — interpreted structurally, T7 probes target effective mention frequency below what current pretraining corpora preserve.

## Knowledge fingerprinting

Combining rare-fact Jaccard overlap with *hallucination similarity* (rate at which two models produce the same wrong answer on rare facts) yields a training-free instrument that distinguishes weight-sharing siblings, post-training lineages, and full retrains across closed-vendor families. The interesting thing here is that *wrong* answers on rare facts are a stronger fingerprint than right ones: two models that learned the same wrong fact almost certainly share training data.

## Connections

- [[mixture-of-experts]] — updated with the total-vs-active capacity result
- [[factual-capacity-scaling]] — the standalone concept that survives outside this paper
- [[thinking-mode-rule-erosion]] — updated with IKP thinking-mode evidence
- [[pine-ai]] — Bojie Li's organization
- [[deepseek]] — V4 Pro shows up as the open-weight ceiling at T6
- [[chatgpt-5-5-pro-mathematical-research]] — Gowers et al on GPT-5.5 Pro; IKP independently confirms 5.5 as the genuine scale-up jump from the 5.x point-release cluster