How much does distillation really matter for Chinese LLMs?

Nathan Lambert published this on interconnects on 24 February 2026, the day Anthropic accused three Chinese labs of "industrial-scale" distillation campaigns against Claude. The post tries to answer one question before the politics: how much performance does a Chinese lab actually gain by training on outputs from an American model? His answer is that it helps, sometimes a lot for a specific capability, but it is not what keeps these labs near the frontier. It is the technical half of a pair. The policy half came in May as the-distillation-panic.

What the word means now

Lambert separates the two meanings. Knowledge distillation in the Hinton, Vinyals and Dean (2015) sense trains a student to match a teacher's probability distribution. That is impossible through an API, because the API does not return the distribution. What people call distillation today is synthetic data: prompt a stronger model, keep its outputs, and train your model to predict them. Lambert calls synthetic data arguably the most useful day-to-day tool an AI researcher has, and most of it is mundane. A lab spends many tokens from a strong model to clean and refine existing data, a lot of compute for a few good training tokens.

The accusations before Anthropic's

The loudest case until then centered on DeepSeek R1, released in early 2025. OpenAI accused DeepSeek of extracting reasoning traces by jailbreaking its API, since those traces are hidden by default; the Bloomberg report Lambert links is from 12 February 2026. Lambert suspects fear of the same thing explains why Gemini switched from showing its reasoning to hiding it, even though early reasoning research (the s1 paper) had been built on Gemini's visible traces.

Reading Anthropic's numbers

Anthropic's post counted over 16 million exchanges through about 24,000 fraudulent accounts. It also called distillation "a widely used and legitimate training method" that frontier labs apply to their own models. Anthropic did not say whether the traffic came through the API, the chat app or claude-code.

Lambert takes the three labs one at a time. DeepSeek is listed first because it is the name Americans know, but its share was small: about 150,000 exchanges covering reasoning, rubric grading (Claude acting as a reward model) and censorship-safe answers to sensitive prompts. At training scale that is an experiment, probably one small team, and Lambert calls its effect on DeepSeek's next model negligible. Using Claude as an online RL grader is possible in principle but very unlikely over access spread across so many accounts.

Moonshot (3.4 million exchanges, aimed at agentic reasoning, tool use, coding, computer use and vision) and MiniMax (13 million, aimed at agentic coding and orchestration) are more substantial. At 10-25K tokens per exchange, the two together come to roughly 150-400 billion tokens. For comparison, Olmo 3's whole SFT set was about 20 billion tokens, and a tenfold larger set built this way would be reasonable. Lambert thinks the choice of teacher makes sense: Claude Opus 4.6 had agentic navigation no other model matched, so training on it was worth trying, and that edge would shrink within months, the way math is now something many models can teach.

Why volume is not the same as uplift

Token counts are a crude measure. Putting another model's outputs into a training pipeline is a research problem, and students sometimes get worse on a given teacher's data for reasons that are hard to predict. Lambert assumes Chinese labs are good at this problem because GPUs are their constraint. For an organization with more money than it can spend on compute, API tokens are a way to buy compute, and getting access to a "banned" API is much easier than smuggling and installing tens of thousands of GPUs. He argues Western labs do the same thing. Olmo 3's synthetic data came from millions of GPU hours on the Frontier supercomputer and Azure credits through NAIRR, compute Ai2 did not own.

He also disputes two popular claims. Chinese labs are not dramatically more efficient than Western ones; they are a little more efficient out of necessity, draw on similar talent, and approach benchmarks in a way that makes them look closer than they are (see open-models-in-perpetual-catch-up and open-closed-model-gap). And distillation does not cap a student below its teacher. Claude Sonnet 3.5 was briefly Anthropic's flagship ahead of Opus, largely because it was well distilled from internal Opus checkpoints. Saying Chinese models can never pass the American models they distill from is, in his comparison, like saying Sonnet could never beat Opus.

The RL argument

The biggest reason he expected the story to fade is reinforcement learning. The best models now need RL at scale, and RL needs on-policy generation from the model being trained. Generation is most of RL's compute bill and cannot be outsourced to someone else's model. A teacher can help craft and filter prompts, but the student still has to do the expensive part itself. The published technical reports from DeepSeek, Moonshot, Zhipu and others show strong RL infrastructure despite compute shortages. Lambert returned to this argument in July, in how-distillation-is-used-today.

Politics

Terms of service have banned using API outputs for "competitive purposes" for years. Academics and US open-model builders argued about the clause in 2022 and 2023, as in frontiers-in-synthetic-data, and stopped worrying late in 2024 because nobody enforced it against small players. Lambert sees Anthropic's disclosure as another step in rising US-China tension. His view is that distillation is much harder to stop than GPU shipments, and that restricting GPU sales would matter far more. His advice to Anthropic is to pick its battles: any model served through an API will be used to train variants of itself, and the only complete defense is to keep the model inside first-party products. He does not expect any leading lab to give up its API.

Where later evidence moved the picture

The post measures DeepSeek by the 150,000 exchanges Anthropic disclosed in February. Anthropic's September report (anthropic-threat-report-september-2026) attributes over 12.1 million exchanges to DeepSeek over 14 days in July 2026, and 151 million to Alibaba between May and July, including chain-of-thought extraction that Anthropic says went into Qwen 3.5-3.7. That report also claims distillation gives "significant uplift" with fewer exchanges than the campaigns harvested. The size of the operations therefore grew well past what this post assessed, while Lambert's estimate of their marginal effect rests on reasoning about RL that the later report does not directly test. The llm-distillation page follows the disagreement.