OLMo: Accelerating the Science of Language Models

OLMo is the first model in the Allen Institute for AI's olmo family. The report, by Dirk Groeneveld, Iz Beltagy, Pete Walsh and about forty co-authors including Nathan Lambert, appeared in February 2024 (arXiv 2402.00838). Its argument is short: as language models became commercially valuable, the strongest ones closed off, and the details a scientist needs (training data, architecture choices, how development went) became undisclosed. OLMo is AI2's attempt at a model that is competitive with the open-weight 7B models of the day while releasing everything behind it.

Degrees of openness

The paper places itself on a scale of existing releases. Mixtral 8x7B shipped weights and a short report. LLaMA came with detailed adaptation instructions. MosaicML's MPT described its dataset distribution but did not release the data. Falcon released part of its pretraining data. The most open efforts were Pythia and BLOOM, which released code, checkpoints and data. LLM360 is named as a contemporary with the same goals. OLMo's claim is that it narrows the gap between that fully open tier and the capability of models like Llama 2.

What AI2 released covers the whole pipeline. For pretraining: training and modeling code, weights for OLMo-7B, a 7B "twin" trained on different hardware, and OLMo-1B, each with over 500 intermediate checkpoints at 1,000-step intervals published as HuggingFace revisions, plus the complete Weights & Biases training metrics. For data: the full Dolma corpus, the code that builds it, the WIMBD dataset-analysis tool, and tools that reconstruct the training order and show which data each step saw. For adaptation: the instruction and preference training code and data, and the SFT and SFT+DPO model weights. For evaluation: the Catwalk and Paloma frameworks and the Tülu evaluation suite. Code and weights are Apache 2.0. This is the practical meaning of "truly open" in the paper, and the property a weights-only release like Mixtral or Llama cannot offer: someone else can reproduce the run, inspect what the model saw, or retrain a variant.

Models and training

The release has four 7B variants, differing in architecture, optimizer and training hardware, and one 1B model. All were trained on at least 2T tokens, one epoch over a 2T-token sample of Dolma, and some went into a second epoch with a different shuffle. The 7B checkpoint used for evaluation was trained to 2.46T tokens, then tuned for another 1,000 steps with the learning rate decayed linearly to zero, which gave a visible jump on most tasks in the last step of the training curve. The table of per-size layer counts, widths and learning rates (Table 1) did not survive extraction in the local copy of the source.

The architecture is a decoder-only transformer with the changes LLaMA, PaLM and Falcon had made standard: no bias terms (for stability), SwiGLU activations with a hidden size of 11,008 at 7B, and rotary positional embeddings. One choice was unusual: a non-parametric layer norm with no learned gain or bias, picked over parametric layer norm and RMSNorm as "the safest option" and the fastest. The tokenizer is a modified GPT-NeoX-20B BPE with extra tokens for masking personal information; the 50,280-token vocabulary sits in a 50,304-row embedding matrix, rounded up to a multiple of 128 for throughput.

Training used PyTorch FSDP with ZeRO sharding, bfloat16 mixed precision with softmax kept in full precision, a global batch of about 4M tokens (2048 sequences of 2048), AdamW with a 5,000-step warmup (about 21B tokens), linear decay to a tenth of peak, and gradient clipping at 1.0. Documents are concatenated with an EOS token and chunked into 2048-token instances, shuffled identically for every run, so the exact batch composition can be rebuilt from the released artifacts.

AI2 trained on two clusters to show the code runs on both vendors: the LUMI supercomputer, up to 256 nodes of four AMD MI250X each, and MosaicML (Databricks), 27 nodes of eight 40GB NVIDIA A100 each. Both reached nearly identical evaluation results by 2T tokens. The appendix estimates 239 MWh to pretrain the 7B models and 69.78 tCO2eq of emissions, counting LUMI's hydroelectric power as zero-carbon per LUMI's official data (at an intensity of 0.024 kg/kWh for hydro, the LUMI run would come to 3.54 tCO2eq), and calls the figures lower bounds because debugging, tuning and downtime are excluded.

Dolma

Pretraining data was the least open part of the model world in 2024, and the paper treats Dolma as a release in its own right. It is a multi-source corpus of trillions of tokens drawn from sources that are common in LLM pretraining and accessible to the public. The pipeline runs language filtering, quality filtering, content filtering, deduplication, multi-source mixing and tokenization, and documents from each source stay separate both during curation and in the release. Details are deferred to the separate Dolma report; the composition table in this paper is another casualty of extraction. The data curation toolkit is open. The later collapse in what web data is licensed for this use is covered in consent-in-crisis-ai-data-commons.

Evaluation

Evaluation happens in two modes. In-loop, a downstream suite runs every 1,000 steps (about 4B tokens) and drives decisions on architecture, initialization, optimizers, schedules and data mixture. Offline, Catwalk scores final checkpoints on eight zero-shot tasks (ARC easy and challenge, BoolQ, OpenBookQA, SciQ, HellaSwag, PIQA, WinoGrande), with the likelihood normalization chosen per task. Against LLaMA-7B, Llama-2-7B, MPT-7B, Pythia-6.9B, Falcon-7B and RPJ-INCITE-7B, OLMo-7B is "competitive" on aggregate. The per-task numbers are in a table that did not come through in extraction.

Perplexity evaluation uses Paloma, 585 text domains from 18 sources, reported in bits per byte so models with different vocabularies compare fairly. OLMo-7B is described as the largest model with explicit decontamination against Paloma: any pretraining document containing paragraphs from Paloma's evaluation data was removed. The fit tracks the training distribution closely. OLMo, with 88.8% Common Crawl data, beats every other model on C4 but trails on WikiText-103 and the M2D2 academic and Wikipedia sets, and MPT-7B improves fastest overall, perhaps because 27% of its data is non-Common-Crawl against OLMo's 11.2%. The authors draw a general point from this: curated text like Wikipedia and arXiv is scarcer than scraped web text, so keeping sample efficiency on it will get harder as corpora grow. On Dolma's 100 programming languages OLMo is far ahead of everyone, which the paper attributes partly to identical code post-processing and partly to contamination Paloma cannot remove from code.

Adaptation

To show OLMo works as a base model, AI2 applied the Tülu 2 recipe: instruction finetuning on a mix of distilled and human-written data (3 epochs at learning rate 2e-6), then Direct Preference Optimization on the UltraFeedback preference set (3 epochs, 5e-7, beta 0.1). Tuning raised MMLU "by a wide margin" and improved ToxiGen and TruthfulQA, most after DPO. OLMo+SFT+DPO beats most other 7B chat models but trails Tülu 2 on Llama 2. The authors offer two reasons: Llama 2 is reported to be contaminated with MMLU test data, and the Tülu mix was designed for Llama. The limitations section adds that the mix relies on data distilled from other models, a dependence AI2 said it wanted to reduce.

What the report admits

The paper lists what it did not do. Data is English only. The page limit left out logs of diverged or failed runs. Most downstream tasks do not look like chatbot use, and evaluations are noisy. The ethics statement makes the openness argument directly: releasing lets others build on the model instead of retraining it at environmental cost, and since permissively licensed comparable models already existed, a stricter license would not have removed the risk from the field. The conclusion notes that a later OLMo 1.7 update, with better data and training, raised MMLU by 24 points to 52%, which is the gap olmo-2-furious set out to close properly.