#distillation
Wiki 14
- 6 months to live for open models Lambert predicts a US ban or delay on frontier open weights by early 2027, calls Anthropic's distillation campaign regulatory capture, and proposes off-ramps
- Detecting and countering misuse of AI: September 2026 Anthropic's threat report on Claude misuse from Dec 2025 to Aug 2026, led by seven Chinese labs said to distill Claude and pass it off as their own models
- Frontiers in synthetic data Lambert's 2024 notes on synthetic data in post-training, from SFT on GPT-4 outputs to Gemini Flash being distilled from Pro
- GLM-5.3: How Chinese labs keep stride with the frontier Lambert on GLM-5.3 and why Chinese labs match US models without relying mainly on distillation, plus Z.ai's staged release for cyber capabilities
- How distillation is used today and what performance uplift it gives to open models Lambert's July 2026 note that distilled data seeds SFT for Chinese labs but matters less as RL grows, written against a Stratechery claim
- How much does distillation really matter for Chinese LLMs? Lambert reads Anthropic's February 2026 disclosure against DeepSeek, Moonshot and MiniMax and argues distillation helps but is not decisive
- LLM Distillation Training one LLM on another's outputs, from SFT on generated text to on-policy KD, and the 2025-2026 fight over Chinese labs distilling US models
- Open models in perpetual catch-up Nathan Lambert on why the roughly six-month gap between open and closed models holds steady, plus trends in adoption, specialization and China
- Open-Source AI & Open Models Reading List Nathan Lambert's annotated list of the best writing on open models — why they exist, why China leads, the gap, cyber risk and distillation
- Reasoning Prefills on Open Models, v1.1 A reasoning-prefill test where Qwen3.8 follows GPT-5.5 Pro's trace far more than other open models, read as a sign of GPT distillation
- Some Simple Economics of Open versus Closed AI Christian Catalini's a16z essay using innovation economics to argue open weights change where AI investment goes and who profits, not how much
- Stealing Reasoning Traces from Proprietary LLM APIs Encrypted chain-of-thought blocks are replayable across sessions, users and models, so a cheap sibling model will decode a frontier model's hidden reasoning
- Synthetic Data & Distillation | RLHF and Post-Training Book by Nathan Lambert Chapter of Lambert's RLHF book on synthetic data, from SFT distillation and on-policy KD to AI feedback, Constitutional AI and rubrics
- The distillation panic Lambert argues "distillation attacks" wrongly brands a standard training technique and warns US policy could end up banning Chinese open weights
Toolbox 1
- needle 26M-parameter function-calling model distilled from Gemini 3.1 for phones, watches and glasses