# Agent readiness

Agent readiness is the set of site-side choices that make a web property usable by AI agents and crawlers — not just indexable. It's a category that didn't exist on most web-platform checklists until [[specification-website]] put it next to SEO and accessibility in 2026.

The framing borrows the SEO and a11y vocabulary — Required / Recommended / Optional / Avoid — and applies it to a stack of mostly-emerging conventions for agent discovery, machine-readable content, declared crawl policy, and crypto-verified bot identity.

## The pieces

**Baseline.** Stable URLs, JSON-LD with schema.org types, alternate data formats (JSON / RSS / markdown) alongside HTML, and an HTTP `Link` header advertising the discovery files so an agent that never parses HTML can still find them.

**Discovery.** [[llms-txt|llms.txt]] gives a curated markdown index; `/llms-full.txt` concatenates the full markdown content. Markdown source is exposed at predictable URLs (typically a `.md` suffix on the canonical URL), so agents pull text directly instead of stripping HTML.

**Declared policy.** Per-agent allow/disallow in robots.txt using vendor-named user agents (GPTBot, ClaudeBot, etc.). The emerging **Content-Signal** robots.txt extension lets a site declare separately whether AI crawlers may *search*, *ingest*, or *train* on each path — an IETF AI Preferences / IAB Tech Lab proposal.

**Identity.** **Web Bot Auth** lets a bot prove who it is by signing each HTTP request with a key it controls (RFC 9421 HTTP Message Signatures). Sites can then allow or block specific bots without IP allow-lists, user-agent strings, or guesswork.

**Tool surfaces.** [[mcp-vs-skills|MCP]] for queryable tool discovery, A2A agent cards at `/.well-known/agent-card.json` for cross-agent delegation, **NLWeb** advertising a conversational `/ask` endpoint via `rel="nlweb"`, and **WebMCP** (`navigator.modelContext`) for in-browser agent tools without server-side plumbing.

**DNS-layer.** **DNS-AID** publishes SVCB/HTTPS records under `_agents.example.com` so agents discover services before any HTTP round-trip; pairs with DNSSEC for authenticated answers.

## Why it's a coherent category

The shape repeats across all of these: drop a file at a fixed well-known path or DNS record, advertise it via a `Link` header or DNS record, let the agent find it without parsing HTML. The same pattern that gave the web `robots.txt`, `sitemap.xml`, and `apple-app-site-association` is being extended for agent use.

The newer half of the category (Content-Signal, Web Bot Auth, A2A, NLWeb, WebMCP, DNS-AID) points at where this is going: site-side declarations of *what* AI access is allowed, plus crypto-verified bot identity so the allow/deny decision doesn't hinge on a forgeable user-agent string.

## Related

- [[llms-txt]] — the most adopted single piece of agent readiness today
- [[supply-chain-security]] — adjacent: tells you who trusted whom, but doesn't help an agent discover what a site exposes
- [[stable-urls-as-public-api]] — would be a natural extraction if/when we ingest more on URL hygiene

## Open page candidates

The following are mentioned across the wiki but don't have their own pages yet — extraction targets when they show up again:

- Web Bot Auth (RFC 9421)
- Content-Signal directives in robots.txt
- A2A agent cards
- NLWeb and WebMCP
- DNS-AID and Schemamap