# Olmo

Olmo is the language model family of the Allen Institute for AI (AI2, styled Ai2). Every release comes with the training data, training and evaluation code, intermediate checkpoints and training logs, not only the weights, under Apache 2.0 where the data allows it. AI2 calls this "fully open" or "truly open". The name was written OLMo, for Open Language Model, until the third generation became Olmo 3.

## Lineage

| Release | Date | Sizes | Tokens | What changed |
| --- | --- | --- | --- | --- |
| [[olmo-accelerating-the-science-of-language-models\|OLMo]] | Feb 2024 | 1B, 7B | 2T to 2.46T | First release; Dolma corpus; Tülu 2 SFT and DPO |
| OLMo 1.7 / OLMo-0424 | Apr 2024 | 7B | not in the sources | Better data mix and curriculum; MMLU up 24 points to 52% |
| [[olmo-2-furious\|OLMo 2]] | Nov 2024 to Mar 2025 | 1B, 7B, 13B, 32B | 4T to 6.6T | Stability fixes; Dolmino mid-training; Tülu 3 with RLVR |
| [[olmo-3-technical-report\|Olmo 3]] and 3.1 | Oct to Dec 2025 | 7B, 32B | about 6.05T to 6.2T | Reasoning, instruct and RL-Zero variants; 65K context; Dolma 3 |

The trajectory is from proving that a fully open model can exist to competing with open-weight models on compute efficiency. The first OLMo was "competitive" with Llama 2 7B on commonsense tasks and trailed Tülu 2 on Llama 2 after instruction tuning. OLMo 2 placed its base models on the frontier of benchmark score per training FLOP against Llama 3.1, Qwen 2.5 and Gemma 2. Olmo 3.1 Think 32B comes within a few points of Qwen 3 32B on reasoning benchmarks after about six times fewer tokens, while still trailing it on knowledge and coding. Compute grew with it: OLMo trained on LUMI's AMD GPUs and a MosaicML A100 cluster, OLMo 2 mostly on AI2's own 1,024-H100 Jupiter cluster, and Olmo 3 32B needed about 56 days on 1,024 H100s from pretraining to the first Think checkpoint. A sibling mixture-of-experts model, OLMoE, shares OLMo 2's pretraining data (see [[mixture-of-experts]]).

The architecture has stayed a dense decoder-only transformer throughout, and the changes are mostly about keeping training stable at scale: OLMo's non-parametric layer norm gave way to RMSNorm applied after each sublayer, QK-norm, z-loss and a new initialization in OLMo 2, and Olmo 3 added sliding-window attention on three of every four layers.

## What "fully open" means in practice

The difference from an open-weight release is what a researcher can do. With weights alone, as with Llama or Mixtral, one can finetune and probe the finished model. With OLMo's artifacts one can also rebuild the exact order of training batches, see which data a checkpoint had seen at any step, retrain part of a run with changed data, check whether a benchmark leaked into training, or start post-training from any intermediate stage. Each report widens the release. The first OLMo shipped weights, 500+ checkpoints, Dolma, the data tools, the Weights & Biases logs and the adaptation code. OLMo 2 added the full mid-training mixes and a new training codebase. Olmo 3 calls the unit of release the "model flow": the mixes actually trained on, the larger source pools they were sampled from (9T, 2T and 640B tokens for its three base stages), every stage's checkpoint, and the decontamination tooling.

Staying fully open costs something that shows up in all three reports. Data has to be licensed for redistribution, which rules out synthetic sets generated under the Llama license; Olmo 3 regenerated several of them with Qwen3. AI2 has crawled its own science PDFs while respecting robots.txt, as the web itself closes to AI crawlers, a trend covered in [[consent-in-crisis-ai-data-commons]]. And results have to hold up to outside checking, which is why the reports keep held-out benchmark sets, publish their decontamination methods, and admit things like the GSM8K problems used to steer OLMo 2's math data.

## Relation to Pythia

EleutherAI's [[pythia-suite-for-analyzing-llms|Pythia]] (2023) is the predecessor in spirit. It set the standard that the OLMo reports measure themselves against: public data, a fixed and reconstructible training order, and dense checkpoints, so that training dynamics can be studied. The first OLMo paper names Pythia and BLOOM as the most open models before it, compares its checkpoints to Pythia-6.9B, and started from a modified version of the GPT-NeoX tokenizer that Pythia also used. The difference is the goal. Pythia deliberately gave up performance for experimental control and trained every model on 300B tokens; OLMo tries to be both open and competitive, which meant training on trillions of tokens and building a post-training recipe.

## Nathan Lambert

Nathan Lambert, who writes [[interconnects]], joined AI2 in October 2023 and is a co-author of all three OLMo reports, a core contributor to OLMo 2 and Olmo 3, and lead author of the Tülu 3 post-training recipe that OLMo 2 and Olmo 3 build on. In [[why-i-build-open-language-models]] (October 2024) he calls his job "white rice research": models that are "very solid, but maybe not quite beating Meta's Llamas", documented at every step so other groups can pick up unfinished ideas, and best understood "as research infrastructure more than just standalone artifacts". The same essay admits AI2 was then "up to years behind" closed labs in RLHF and could use ten times its compute. His [[atom-project-american-truly-open-models|ATOM Project]] (August 2025) lists Ai2's Olmo first among American open model builders as the "open-source leader" and calls Olmo 3 32B Think "the best fully open reasoning model", and argues the US needs several labs with 10,000+ GPUs training such models. Both are the author describing his own project, which is also true of the [[open-source-ai-reading-list]] entries that point at these reports.

## Summary pages

- [[pythia-suite-for-analyzing-llms]] — EleutherAI's 2023 suite of 16 models trained in one fixed data order, built for training-dynamics research
- [[olmo-accelerating-the-science-of-language-models]] — the first OLMo, 1B and 7B with Dolma, code, logs and checkpoints
- [[olmo-2-furious]] — OLMo 2 at 7B to 32B, stability fixes, Dolmino mid-training and Tülu 3 post-training
- [[olmo-3-technical-report]] — Olmo 3 Base, Think, Instruct and RL-Zero, the "model flow" release and OlmoRL