# Consent in Crisis: The Rapid Decline of the AI Data Commons

Shayne Longpre, Robert Mahari, Ariel Lee and Campbell Lund led this Data Provenance Initiative paper (arXiv 2407.14933, July 2024), with more than forty contributors and advisors including Stella Biderman, Sara Hooker, Daphne Ippolito and Luis Villa. It is the first large-scale longitudinal audit of consent signals on the websites underneath open AI training corpora. It covers how often sites block AI crawlers, how badly they express that, and what the blocked content was. Nathan Lambert's [[open-source-ai-reading-list]] includes it as the documentation of "the mass reduction in open data, which is a crucial factor that has hampered truly open AI research". It is the only piece in the list's safety section that is about data rather than misuse.

The data ends in April 2024, and every figure on this page is from that date. Newer mechanisms for expressing crawl preferences are covered elsewhere in the vault and noted below.

## What was audited

The paper audits the websites that datasets were built from, not the datasets. It took three open, permissively licensed corpora derived from Common Crawl: C4 (April 2019 crawl, about 15.9 million domains), RefinedWeb (2008 to February 2023, 33.2 million) and Dolma (May 2020 to June 2023, 45.2 million). From each it took the top 2,000 domains by token count, 3,950 domains in total, called the head sample. This is the "largest, most actively maintained, and critical" material. A random 10,000 domains from the intersection of all three corpora served as the tail sample, 2,000 of them human-annotated. Annotators, paid $25-30 an hour, labelled content types, site purpose, paywalls, ads and terms of service.

For each domain the team pulled monthly snapshots of the homepage, robots.txt and terms-of-service page from the Wayback Machine, January 2016 to April 2024. Robots.txt was classified for crawlers of seven "AI organizations": Google, OpenAI, Anthropic, Cohere and Meta, plus Common Crawl and the Internet Archive, whose archives get repurposed for training. A token counted as "restricted" if its domain fully blocked at least one of them. Terms of service were classified by an LLM pipeline. Trends were forecast a year ahead with SARIMA models.

## Restrictions rose sharply in one year

In mid-2023 about 1% of tokens in the three corpora came from domains that fully restricted an AI crawler. By April 2024 that was 5-7%. In the head sample it went from under 3% to 20-33%, a relative rise of over 500% for the full C4 and RefinedWeb corpora and over 1,000% for their heads. Those numbers count only full blocks; partial restrictions add more. The abstract gives the headline versions: 5%+ of all C4 tokens and 28%+ of the most critical C4 sources are now fully restricted by robots.txt, and 45% of C4 carries some terms-of-service restriction.

The timing points to the new user agents. The shift came right after OpenAI's GPTBot and Google-Extended appeared in 2023, mostly from sites that had previously blocked a few directories moving to a full block. News sites drove most of it: in the head sample nearly 45% of news tokens were fully blocked in April 2024, against 3% in 2023. Forums and social media followed. Terms of service moved more slowly, from no terms page at all toward anti-crawling, non-commercial and non-compete clauses. Terms that name generative AI explicitly appeared broadly only in 2024. Crawlers mostly ignore terms of service in practice, but terms give sites a legal route.

The forecast for April 2025 was another 2-4% of full-corpus tokens and 7-11% of head tokens blocked by robots.txt, and a 6-10% rise in terms restrictions.

## Consent is expressed badly

Robots.txt dates from 1995 and was codified in 2022. It lets a site say *who* may crawl, not *what for*. The paper finds several ways this breaks down.

Blocking is uneven across developers. In the head of C4, OpenAI's crawler was blocked on 25.9% of tokens, Anthropic's and Common Crawl's on 13.3%, Google-Extended on 9.8%, Cohere's on 4.9%, Meta's on 4.1%, the Internet Archive's on 3.2% and Google Search on 1.0%. Conditioned on a site blocking any AI organization, OpenAI was blocked 91.5% of the time and Meta 52.2%. The authors read this as site owners not knowing the full list of agents to block, which in practice favours lesser-known developers.

Sites also block agents nobody operates. 4.5% of websites disallowed `anthropic-ai` or `Claude-Web`, user agents Anthropic does not recognize as its own, while leaving its documented crawler `ClaudeBot` allowed.

And the two signals contradict each other. In the head of C4, 35.1% of tokens came from sites with anti-crawling terms but no robots.txt, and 20.3% from sites with a restrictive robots.txt and no terms. Sites whose terms only forbid commercial use often block crawlers outright, which shuts out academic crawlers as a side effect.

Some developers appear to route around opt-outs by treating crawling at inference time as different from crawling for training. OpenAI is one of the few to separate the two, with GPTBot for training and ChatGPT-User for live browsing. The paper expects creators to see unregistered retrieval crawlers as violating the spirit of an opt-out.

## What the blocked content is

The head and the tail of the web differ sharply. The head is 72.9% news, forums and encyclopedias, while the random tail is 97% personal and organization sites, blogs and e-commerce. The head is also far more monetized, with ads (+47.5 percentage points) and paywalls (+24.1), and far more multimodal, with video (+39.8) and audio (+38.4). It carries the most restrictions: robots.txt +22.5, terms +35.3. A crawler that respects the signals loses the freshest, best-curated, most multimodal sources first, and its data drifts toward old content and the long tail of corporate pages.

The paper then compares training data with actual chatbot use, using 1,000 conversations from WildChat (a public log of GPT-3.5 and GPT-4 conversations collected through a Hugging Face Space), labelled by GPT-4o. News is nearly 40% of head-C4 tokens but under 1% of conversations. Creative writing and role-play are over 30% of conversations and rare in web data. Sexual role-play is 12% of conversations, while sensitive content is under 1% of head domains, because every audited corpus filters it out. The authors mention a possible fair-use implication, that chatbots may not compete with most of their training sources, and immediately warn against reading too much into it. WildChat's users are probably more technical than average, and the taxonomy is coarse.

## The argument

If the restrictions are respected, as many developers say they are, or enforced, then the amount, diversity, freshness and representativeness of good pretraining data will shrink, and data scaling laws will be affected. The cost does not land only on commercial labs. Sites that find it too tedious to list every AI agent block everything with a wildcard, or block Common Crawl and the Internet Archive to keep data away from whoever downloads their archives. Those archives serve academic and non-commercial research far beyond AI; Common Crawl is cited in more than 10,000 papers. The paper calls these groups "secondary victims" of a fight between creators and commercial developers. It expects small publishers to retreat behind logins and paywalls.

The proposed fix is a protocol that expresses *use* rather than *identity*: allowing search, or non-commercial AI, or AI that attributes its sources, plus time-bounded blocks so news sites can protect freshness. The vault's [[agent-readiness]] page records one proposal of that shape, the Content-Signal robots.txt extension, which lets a site separately allow *search*, *ingest* or *train*. The paper's ethics statement is careful. Site administrators are imperfect stand-ins for copyright holders, especially on user-content platforms. The authors do not say which signals should be binding, and they note the competing argument for a "right to read and learn" from the public web.

## Why it is on an open-models reading list

Lambert's framing is about open research, and the paper's ethics statement points the same way: the rules that emerge around consent will decide "the types of organizations that are able to acquire sufficient data to compete". A fully open project such as [[olmo]] has to publish its training corpus, so every restriction it honours, or fails to honour, is visible to anyone who looks. Dolma, one of the three audited corpora, is OLMo's. [[societal-impact-of-open-foundation-models]], with Longpre and Kevin Klyman among its authors, had argued that open data is a precondition for much of the scientific value of open models, and this audit measures that precondition getting worse. [[why-i-build-open-language-models]] gives the builder's side of the same concern.

The other side of the dispute appears in [[llm-enshittification]], where a Gentoo developer describes LLM scrapers ignoring robots.txt and hitting Gentoo's Bugzilla like a "constant DDoS attack". The paper's quantitative picture and that first-hand account agree: the sites doing the blocking are reacting to real crawler behaviour. The paper deliberately does not rule on whether the blocking is justified.