How distillation is used today and what performance uplift it gives to open models

This short post from July 2026 appeared on Nathan Lambert's personal Substack, not on interconnects. It reads like working notes: a quoted tweet, then a list of where he stands. The saved copy is about 750 words and ends with a reading list and Substack's comment footer, so it is the whole post and not a truncated preview. The rough phrasing ("Then… where I stand", "Getting to the end of it") is in the original.

The trigger

Ben Thompson wrote on Stratechery that Chinese labs distill during RL and that this is becoming more important to model performance. Lambert disagreed on X: the Chinese labs are not using Fable, the strongest model, as a teacher during RL. "That's not how distillation works." Graders in RL are messy, so the lift would be small, and nobody can afford to run Fable at that volume. (Anthropic's September report, anthropic-threat-report-september-2026, calls Fable its top generally available model.)

Where distillation actually sits

Anthropic had said DeepSeek and others used its models in an RL-shaped data pipeline. Lambert's reading is that those were small sample counts, probably to initialize an internal model or run an experiment, and that "RL-shaped" does not mean large effect.

The step where distillation matters is SFT and midtraining, which he treats as one stage for seeding reasoning behavior. Next-token training teaches a model recurring clusters of tokens, and that is why some Chinese models say they are Claude. The data for this stage comes from getting around an API's intended behavior to obtain reasoning tokens. Without the raw reasoning the data would be much harder to train on, and would probably not carry the "I'm Claude" habit either.

A reasoning SFT set is on the order of a million prompts with high-quality completions. Prompts are often the hardest part, especially prompts backed by environments that later RL can use. Distilled SFT is most useful when a model enters a domain where US labs are far ahead. His example is physics on the CritPT benchmark (spelled "CripPT" in the post; the link goes to CritPT): collect prompts, get completions from a frontier model, run an initial SFT.

Why the uplift is shrinking

An initial SFT set is the start of a long process. Reaching the level of GLM-5.2 (glm-5-2-step-change-for-open-agents) or Kimi K3 (kimi-k3) still takes more SFT data generation, filtering, something like rejection sampling, and a lot of RL. On timing alone, Lambert judges that Fable 5 probably had no effect as a teacher for Kimi K3 or GLM-5.2, though it could affect later models.

Post-training is also moving toward multi-teacher on-policy distillation (MOPD) as well as RL. The more steps a recipe has, the harder it is to trace what an early SFT stage contributed. RL now accounts for a much larger share of final performance than before. When SFT on a base model produced most of the post-training gain, distillation mattered more. Lambert expects the trend to continue.

The best teacher is often not the best model

The last point is the one he finds most interesting. In SFT, the strongest model is often not the easiest teacher to fit into a recipe. OpenThoughts and Olmo, the leading fully open reasoning-SFT efforts, had trouble switching to the strongest models as teachers, and the best teacher often turned out to be a smaller, unexpected model. Cutting-edge models may therefore not be what makes distillation work.

He draws two consequences. Strong open-weight models remove much of the reason to use Claude or GPT APIs at all, which weakens the claim that open models are "distillation washing" closed ones. And open models may simply be easier to distill from, because they are easier to modify. The book chapter he links, rlhf-book-synthetic-data-distillation, gives the technical background on on-policy distillation and MOPD.

How it fits the rest of his writing

The post extends the RL argument from how-much-does-distillation-matter-for-chinese-llms (February 2026) with a finer picture of where teacher data enters a pipeline. It came out a few weeks before stealing-reasoning-traces-from-proprietary-llm-apis and Anthropic's September report. That report describes Alibaba converting Opus chain-of-thought into SFT data for Qwen 3.5-3.7, and Xiaomi replaying user sessions through Claude to make data "for both SFT and RL". The SFT-first part matches Lambert's account. The volume, 151 million exchanges from Alibaba alone, is far larger than the "very small numbers of samples" he based his view on. In September, his reading list moved toward more realism about what trace extraction could have done (see llm-distillation).