Map
↑specification.website

Agent readiness

Wiki conceptai-agentsweb-standards ↳ show in map Markdown
title
Agent readiness
type
concept
summary
Site-side choices that make a web property legible to AI agents — stable URLs, machine-readable endpoints, declared crawl policy, verifiable bot identity
parent
specification-website
tags
ai-agents, web-standards
sources
specification-website-checklist
created
2026-05-31
updated
2026-05-31

Agent readiness is the set of site-side choices that make a web property usable by AI agents and crawlers — not just indexable. It's a category that didn't exist on most web-platform checklists until specification-website put it next to SEO and accessibility in 2026.

The framing borrows the SEO and a11y vocabulary — Required / Recommended / Optional / Avoid — and applies it to a stack of mostly-emerging conventions for agent discovery, machine-readable content, declared crawl policy, and crypto-verified bot identity.

The pieces

Baseline. Stable URLs, JSON-LD with schema.org types, alternate data formats (JSON / RSS / markdown) alongside HTML, and an HTTP Link header advertising the discovery files so an agent that never parses HTML can still find them.

Discovery. llms.txt gives a curated markdown index; /llms-full.txt concatenates the full markdown content. Markdown source is exposed at predictable URLs (typically a .md suffix on the canonical URL), so agents pull text directly instead of stripping HTML.

Declared policy. Per-agent allow/disallow in robots.txt using vendor-named user agents (GPTBot, ClaudeBot, etc.). The emerging Content-Signal robots.txt extension lets a site declare separately whether AI crawlers may search, ingest, or train on each path — an IETF AI Preferences / IAB Tech Lab proposal.

Identity. Web Bot Auth lets a bot prove who it is by signing each HTTP request with a key it controls (RFC 9421 HTTP Message Signatures). Sites can then allow or block specific bots without IP allow-lists, user-agent strings, or guesswork.

Tool surfaces. MCP for queryable tool discovery, A2A agent cards at /.well-known/agent-card.json for cross-agent delegation, NLWeb advertising a conversational /ask endpoint via rel="nlweb", and WebMCP (navigator.modelContext) for in-browser agent tools without server-side plumbing.

DNS-layer. DNS-AID publishes SVCB/HTTPS records under _agents.example.com so agents discover services before any HTTP round-trip; pairs with DNSSEC for authenticated answers.

Why it's a coherent category

The shape repeats across all of these: drop a file at a fixed well-known path or DNS record, advertise it via a Link header or DNS record, let the agent find it without parsing HTML. The same pattern that gave the web robots.txt, sitemap.xml, and apple-app-site-association is being extended for agent use.

The newer half of the category (Content-Signal, Web Bot Auth, A2A, NLWeb, WebMCP, DNS-AID) points at where this is going: site-side declarations of what AI access is allowed, plus crypto-verified bot identity so the allow/deny decision doesn't hinge on a forgeable user-agent string.

  • llms-txt — the most adopted single piece of agent readiness today
  • supply-chain-security — adjacent: tells you who trusted whom, but doesn't help an agent discover what a site exposes
  • stable-urls-as-public-api — would be a natural extraction if/when we ingest more on URL hygiene

Open page candidates

The following are mentioned across the wiki but don't have their own pages yet — extraction targets when they show up again:

  • Web Bot Auth (RFC 9421)
  • Content-Signal directives in robots.txt
  • A2A agent cards
  • NLWeb and WebMCP
  • DNS-AID and Schemamap
Linked from