llms.txt

A proposed convention for placing a markdown file at /llms.txt on any website to give language models a concise, structured overview of the site's content. Spec lives at llmstxt.org.

The problem it solves

LLMs have finite context windows. Dumping an entire documentation site into a prompt doesn't work — there's too much content, most of it irrelevant to the current task. Sitemaps exist but they're XML lists of URLs with no semantic information. Robots.txt tells crawlers what to avoid, not what to read first.

/llms.txt fills the gap: a human-curated markdown file that tells an LLM "here's what this site is about, and here are the important pages in markdown form."

Format

The file is plain markdown with a specific structure:

1. H1 heading — project or site name (the only required part)
2. Blockquote — brief summary with essential context
3. Body content — detailed project information
4. H2 sections — curated lists of markdown links with descriptions, pointing to the important pages
5. "Optional" section — secondary info that can be dropped if the context window is tight

Sites are expected to provide .md versions of their pages alongside the HTML (e.g., appending .html.md to the URL). This gives the LLM clean text without having to parse HTML.

How agents use it

An agent hitting an unfamiliar site fetches /llms.txt first to understand what's available and where to look. Instead of crawling the whole site or guessing at URL patterns, it gets a curated table of contents with direct links to markdown versions of the key pages.

This is particularly useful for:

• Documentation sites (SDKs, APIs, frameworks)
• Project homepages with scattered technical details
• Any site where an agent needs to find specific information without browsing

Adoption

Integrations exist for VitePress, Docusaurus, Drupal, and other static site generators. Multiple directories catalog which sites have /llms.txt files. Tools like llms_txt2ctx parse the file and expand linked pages into a single context block.

The agent-reading-test benchmark uses /llms.txt as its machine-readable site index, testing whether agents can navigate documentation using this convention.

Relationship to other standards

/llms.txt doesn't replace robots.txt (access control) or sitemap.xml (exhaustive URL list). It sits alongside them with a different purpose: not "what exists" or "what's allowed" but "what's important and how to read it." It's designed for inference-time use, not training data collection.

The specification-website-checklist places /llms.txt as one piece of a broader agent-readiness stack — /llms-full.txt for the full markdown body, markdown source exposed at .md URLs alongside HTML, and the HTTP Link header advertising all three so agents that never parse HTML can still find them.