2 OLMo 2 Furious

OLMo 2 is the second generation of the Allen Institute for AI's olmo models. The report, credited to the OLMo Team with Pete Walsh, Luca Soldaini, Dirk Groeneveld and Kyle Lo as leads and Nathan Lambert among the core contributors, was first posted in December 2024 (arXiv 2501.00656); the version in the vault also covers the 32B model released in March 2025. The family is dense decoder-only models at 7B, 13B and 32B, with a 1B used for recipe development and later released too. Where the first OLMo was mainly a demonstration that a truly open model could exist, OLMo 2's claim is performance per unit of compute: base models on the Pareto frontier of benchmark average against training FLOPs, matching or beating Llama 3.1, Qwen 2.5 and Gemma 2 at their sizes while spending less.

What "fully open" covers here

The introduction draws the line against the open-weight releases of 2024 (Llama 3, Qwen 2, Gemma, Mistral, Phi and others): those weights are "only the final artifacts" of a pipeline, and are not enough to study how models come to behave as they do. The fully open camp it places itself in includes Pythia, the first OLMo, Amber, DCLM, MAP Neo and SmolLM. OLMo 2 ships the weights for every size and stage (base, SFT, DPO and final instruct), all pretraining and mid-training data, the training code in both the original OLMo repository and the rewritten OLMo-core, the open-instruct post-training code, the OLMES evaluation suite and the Dolma curation tools, Weights & Biases or Comet training logs for 7B, 13B and 32B, and thousands of intermediate checkpoints. Everything is Apache 2.0 where possible, otherwise the most permissive license available.

Models and token counts

All use a 4096-token context. The 32B is the only one with grouped-query attention, a choice the report says was inspired by Qwen 3. The 1B's total is inferred from its 4T pretraining plus one 50B anneal; the report gives no single number for it.

The stability fix list

The largest technical section is about why OLMo-0424, the April 2024 model, could not be scaled. Its runs had frequent loss spikes, often preceded by spikes in gradient norm, plus a slow upward creep in gradient norm across training; bigger models spiked more, and together these pointed to eventual divergence. OLMo 2 changes eight things, and the report measures most of them with a "spike score": the percentage of values more than seven standard deviations from a rolling 1,000-step mean.

Data came first. Batches that spiked often held long repeated n-gram runs (base64 junk, lists of 255, 255, 255). The relationship is not deterministic, since the same sequence can spike one model and not another, or spike under one data order and not after a reshuffle. Removing documents with 32 or more repetitions of any 1-to-13-token n-gram, and masking such sequences out of the loss in the trainer, cut many spikes but did nothing for the slow gradient-norm growth.

Initialization was the biggest single fix. OLMo-0424 scaled each layer's weights down with depth; OLMo 2 draws every parameter from a normal distribution with mean 0 and standard deviation 0.02. In a deliberately unstable test setup with short warmup, the gradient-norm spike score fell from 0.40 to 0.03. The report backs this with a "growth exponent" measuring how activation and gradient norms grow across layers at initialization (closer to zero for the new scheme at every width) and with gradient norms that scale with the square root of width, a property associated with hyperparameters transferring across model sizes.

Architecture changes came as a group. The non-parametric layer norm that was OLMo 1's distinctive choice goes back to RMSNorm; ablations showed no difference, and the original library bugs that motivated it were gone. Normalization moves from the inputs of the attention and MLP sublayers to their outputs, so each block computes h = x + RMSNorm(Attention(x)), and queries and keys get their own RMSNorm (QK-norm) before attention. Neither helps alone; together they took the gradient spike score from 0.108 to 0.069. Z-loss regularization keeps output logits from growing. RoPE theta rises from 10,000 to 500,000, as in Llama 3. On the optimizer side, AdamW's epsilon drops from 1e-5 to PyTorch's default 1e-8, which lets early updates be larger and makes the gradient norm settle quickly and stay lower. Weight decay is turned off for embeddings, because decaying them shrank their norm and inflated early-layer gradients (spike scores 0.16 with decay against 0.092 without).

Flash Attention's fused z-loss and a plain PyTorch implementation agree in the forward pass but diverge in the backward pass, probably from precision differences. The effect on loss and benchmarks was invisible, but AI2 abandoned the fork of the run that had switched implementations and retrained from the point of divergence.

Mid-training and Dolmino

Base training runs in two stages. Pretraining, 90 to 95% of FLOPs, uses OLMo 2 Mix 1124: 3.90T tokens, over 95% of it web text from DCLM-Baseline (3.71T), plus 83B tokens of StarCoder code filtered to repositories with at least 2 stars, 58.6B of peS2o academic papers, 20.8B of arXiv, OpenWebMath and Algebraic Stack at about 12B each, and 3.7B of Wikipedia and Wikibooks. The mix is shared with OLMoE. The cosine learning-rate schedule is set for 5T tokens and truncated: the 7B stops at 4T and the 13B runs to 5T.

Mid-training, 5 to 10% of FLOPs, decays the learning rate linearly to zero on Dolmino Mix 1124, a smaller mix built to add STEM knowledge and patch math. Its high-quality part is 832.6B tokens: DCLM web documents in the top 7% by a FastText classifier that also score 2 or more on the FineWeb-Edu classifier (752B), decontaminated FLAN instruction data, peS2o, Wikipedia, and Stack Exchange Q&A filtered by votes. The math part is 10.7B tokens, much of it synthetic: TinyGSM-MIND (6.5B tokens of code-answered GSM8K-like problems rewritten into natural-language dialogue by Qwen2.5-7B-Instruct), MathCoder2-style synthetic textbooks, 230M tokens of persona-driven problems and solutions from GPT-4o, and 28M tokens of targeted arithmetic and number-swapped GSM8K. Samples of 50B, 100B and 300B tokens are drawn so that filtered web is about half of each; the small sources are repeated up to 4x in the larger samples.

The effect of this stage is the report's headline number. The 7B average across its benchmark suite rises 10.6 points (53.0 to 62.9), passing where the 13B stood after pretraining alone, and GSM8K jumps from 24.1 to 67.5. The 13B gains 10.3 points and goes from 37.3 to 75.1 on GSM8K. The relative gain shrinks with size: 37% at 1B, 12.3% at 32B.

Three findings from building the stage are worth keeping. First, a learning-rate sweep showed that higher peak rates look better early but are overtaken, and that after annealing, runs from 3e-4 to 12e-4 end at nearly the same loss and benchmark scores: a higher rate makes the anneal more effective by exactly as much as it made pretraining worse. The report says this goes against folk wisdom that area under the learning curve matters, and warns that the crossover came well past 200B tokens, so a short sweep would have picked wrong. Second, "microanneals" test a data source cheaply by annealing on a 50/50 mix of that source and web text: 19 of them cost 130B tokens, less than the three 50B anneals used for the final 7B. They showed that a 10% math share gets almost the whole GSM8K gain of a 35% share, that repeating scarce math data 2x helps, and that TinyGSM with code answers hurt GSM8K while the same problems rewritten into prose raised it from 28.5 to 65.5. Third, averaging checkpoints ("souping") from anneals on different data orders matched or beat the best single run on all six mixes tested. The final 7B averages three 50B anneals; the 13B and 32B average three 100B anneals and one 300B anneal.

The report is careful about its evaluation. It separates development benchmarks from a held-out set (AGIEval, GSM8K, MMLU-Pro, TriviaQA) never used for decisions, and admits GSM8K was only partly held out: 200 of its 1,319 problems were used to steer the math mix, and only the other 1,119 are scored. It asks other developers to declare which tasks they monitored.

Post-training

OLMo 2-Instruct applies AI2's Tülu 3 recipe in three stages with two changes: only permissively licensed models generate the preference data, and the RL stage is extended to several rounds. SFT uses about 939K prompts for 7B and 13B and 866K for 1B and 32B, the second mix filtering out synthetic answers that mention a knowledge cutoff (they taught the model to hallucinate cutoffs and open with "As an AI language model") and keeping only math answers that won a 5-sample majority vote. Removing the multilingual data from Tülu 3's mix cost about half a point, so it stayed. OLMo 2 needed much higher SFT learning rates than Llama 3.1 did under the same recipe.

DPO is on-policy: roughly 367K to 378K prompts, responses sampled from the OLMo SFT checkpoints and a pool of 20 other models, rated by GPT-4o for helpfulness, truthfulness, honesty and instruction following, then binarized. The last stage is reinforcement learning with verifiable rewards (RLVR), where the policy is rewarded only when a math answer or a formatting constraint checks out. The 7B and 13B use PPO with the value function initialized from a reward model; the 13B needed two extra rounds, on GSM8K and then MATH alone, after its first pass fell short on math. The 1B and 32B use GRPO, which needs no reward model. The first instruct release had to be retrained after AI2 found the instruct tokenizer had lost the base model's pre-tokenization logic; those models are now labeled "preview".

The results put OLMo 2 13B Instruct (average 63.5) above Llama 3.1 8B Instruct and Tülu 3 8B and close to Qwen 2.5 14B Instruct (65.3). OLMo 2 32B Instruct averages 68.8, level with Qwen 2.5 72B and above GPT-3.5 Turbo (60.5) and GPT-4o mini (65.7) on the same suite, though below Llama 3.3 70B (72.7). Code was not a target skill and is left out of the instruct evaluation.

Infrastructure and cost

The report argues that infrastructure details are part of reproducibility and usually get skipped. Training ran on two AI2 clusters: Jupiter in Austin, 1,024 H100s in 128 nodes with 8x400 Gbps InfiniBand per node and a PUE of 1.2, and Augusta, a 160-node Google Cloud cluster in Iowa (its GPU specification did not survive extraction). The Beaker scheduler moved jobs between them with a one-line change, ran a tensor-multiplication health check before each job, and cordoned bad nodes automatically. Four PyTorch practices are described in enough detail to copy: torch.compile, hunting down host-device synchronizations (with torch.cuda.set_sync_debug_mode("warn")), doing metrics and checkpointing in a separate thread on a GLOO backend because NCCL is not thread safe, and disabling Python's garbage collector so every rank collects at the same step. Pretraining the 7B and 13B used about 391 MWh and emitted about 154 tCO2eq, and for the first time the report estimates water, about 1.1 million liters. The 7B alone is 131 MWh against 1,022 MWh reported for Llama 3.1 8B.

Weak spots the report names

The 1B was hard to scale in tokens. It stays competitive with SmolLM2 but trails the smaller Gemma 2 and Qwen 2.5 bases, and after pretraining alone it could not beat chance on MMLU or ARC-Challenge. The authors guess that below some size, pretraining needs task-shaped data or distillation from a larger model, which is how Gemma 2's small models were built, while noting that the same instruct recipe makes the 1B competitive with Qwen 2.5 1.5B and Gemma 3 1B after all. The learning-rate sweep was not wide enough to find the edges of the plateau. olmo-3-technical-report picks up from here with longer context, reasoning models and a data pipeline rebuilt for the licensing pressure described in consent-in-crisis-ai-data-commons.