Synthetic Data & Distillation | RLHF and Post-Training Book by Nathan Lambert

This is a chapter of a book. Nathan Lambert's Reinforcement Learning from Human Feedback is a post-training textbook published in 2026 and free at rlhfbook.com, where this chapter sits at /c/12-synthetic-data. It refers to other chapters by number: policy gradients in 6, verifiable rewards in 7, KL divergence in 15, evaluation in 16, character training in 17. The open-source-ai-reading-list gives it as the background reading for distillation. It is a survey with equations and citations. It takes no position on the policy fight covered in the-distillation-panic, though Lambert quotes the chapter's definition in that post.

The chapter starts from a historical point. RLHF assumed human data because humans were once the only source of good answers and reliable feedback. Once models got better, synthetic data made experiments cheap, and that cheapness is a large part of why RLHF spread.

Where synthetic data won and where it did not

Model collapse, the idea that training on a model's own outputs narrows the distribution until rare facts vanish, is treated as a real but avoidable failure. It shows up mainly with unfiltered, repetitive, single-model self-training. Mixing in human data, using several teachers, deduplicating and filtering keep a pipeline out of that regime, and the chapter says frontier pipelines should use synthetic data at scale.

Synthetic data became useful with GPT-4-class models. Llama 2 and GPT-3.5-Turbo were not reliable enough to generate or supervise data, and LLM-as-a-judge only started working in the move to GPT-4. Dataset sizes show the change: Alpaca had 52K prompts and about 10M tokens, Tülu 3 about 500M tokens, and OpenThoughts 3 on the order of 10B.

The result differs by data type. For SFT, synthetic data has largely won, and distilled completions beat what most human writers can produce at scale. For preference data, academic results say synthetic data works about as well, yet frontier labs still treat human preferences as a moat. For evaluation, models do the scoring cheaply while humans still write the benchmarks and ground truth. Synthetic data wins where models are more reliable than people. Humans stay at the capability frontier and in setting ground truth.

Two meanings of distillation

The formal meaning comes from Hinton et al. (2015): a student trained on a teacher's soft labels. The colloquial meaning is training on a stronger model's outputs. In post-training the colloquial kind is used in two ways. One is a data engine for the whole pipeline (instruction completions, preference data or Constitutional AI, verification for RL). The other moves a specific skill such as math or code into a smaller model. Labs keep large internal models like Claude Opus or Gemini Ultra partly to generate this data, and open models commonly distill from closed APIs, going back to Zephyr. Curating prompts and filtering responses matter most.

From offline to on-policy knowledge distillation

The most technical section explains why teacher-student distillation came back as reasoning and agentic models took over post-training. The chapter names Qwen3, Xiaomi's MiMo-V2-Flash, Zhipu's GLM-5 and DeepSeek-V4-Pro as models trained with newer forms of it.

Kim and Rush (2016) adapted knowledge distillation to sequences. Word-level KD matches the teacher's next-token distribution at every position. Sequence-level KD replaces the intractable sum over all sequences with one beam-searched teacher output, which reduces to ordinary cross-entropy on text the teacher wrote. The chapter calls this family offline KD because the training text is generated in advance. DistilBERT and TinyBERT belong to it. With a fixed teacher, minimizing cross-entropy is the same as minimizing forward KL from teacher to student, the direction SFT uses.

Offline training causes exposure bias. The student learns on prefixes the teacher wrote but generates from its own prefixes, so one bad token puts it in unfamiliar territory, where it is more likely to make another mistake. The chapter borrows the imitation-learning bound behind DAgger, where error grows as O(εL²) in sequence length L, and presents it as an analogy for LLMs rather than a guarantee. That growth hurts badly at thousands of tokens.

On-policy distillation (OPD) samples from the student and asks the teacher to score the states the student actually reaches, which the DAgger analysis says brings compounding down to O(εL). The objective becomes reverse KL, student to teacher (MiniLLM; Agarwal et al.). Current implementations put it directly inside RL: the per-token advantage is the teacher's log-probability of the sampled token minus the student's, following Thinking Machines' write-up. That gives dense token-level feedback instead of a single sparse reward. Teacher and student usually need the same tokenizer, which is unusual among post-training methods.

Variants

Multi-teacher OPD (MOPD, from the MiMo-V2-Flash report) weights several specialist teachers per prompt, so separate teams can build math, code or other experts that later teach one general model. The chapter says DeepSeek-V4 and MiMo-V2-Flash were trained this way. On-policy self-distillation (OPSD) has a model act as its own teacher using privileged information, such as a verified answer or tool results. Cursor's Composer 2.5, fine-tuned from Kimi K2.5, used a version of it. A judge prompted with a list of common bugs reviews RL trajectories, inserts a hint where it finds a bug, and the distillation loss is computed on the corrected sequence. The chapter's suggested experiment runs SDPO, a related self-distillation method, on a string-reversal toy task on one GPU.

AI feedback, Constitutional AI and rubrics

The second half covers AI feedback. A human preference label costs $1 or more, sometimes over $10 per prompt, against under a cent for frontier-model feedback. Lambert's rule of thumb: human data is high-noise and low-bias, synthetic preference data is low-noise and high-bias, which makes it easy to start with and liable to subtle systematic side effects. Later work routes the hard items to humans and the rest to models. Dedicated judge models (Prometheus, Auto-J, Shepherd, CritiqueLLM) exist but rarely appear in documented recipes, since frontier models are already trained heavily as judges.

Constitutional AI is described as the earliest documented large-scale use of synthetic data in RLHF. A model critiques and revises its answers against written principles to produce SFT data, and picks the better of two answers under a sampled principle to produce preference data. The chapter's related items include OpenAI's Model Spec and deliberative alignment.

The chapter ends with rubrics. These are per-prompt lists of weighted criteria, written by a model, that give near-verifiable rewards for prompts with no checkable answer, and they have extended RL to scientific reasoning, factuality and deep research. Three full rubric-generation prompts are reproduced. The same rubric grading shows up in Anthropic's February 2026 account of DeepSeek using Claude "as a reward model", discussed in how-much-does-distillation-matter-for-chinese-llms.

Source note

The clipped bibliography is incomplete. Entries 5, 11, 16, 19, 30, 39, 40, 45, 65 and 67 are missing, including the references for Qwen3, DeepSeek-V4, Zephyr and Cursor's Composer 2.5, although the text cites all of them. The math is preserved as escaped LaTeX. See llm-distillation for how this chapter's taxonomy connects to the 2026 dispute over Chinese labs.