Agent Reading Test

A benchmark that tests whether AI coding agents can read and extract information from web documentation. Ten pages describe a fictional "DataStream" platform, each targeting a different failure mode from the Agent-Friendly Documentation Spec. The agent fetches the pages, answers specific questions, and self-reports what it found. You compare against the answer key for a score.

How it works

The test lives at agentreadingtest.com. You point your agent at /start/, which lists 10 tasks with URLs. Each task asks a concrete technical question — identify API parameters, document default pool settings, list aggregation types, show SDK initialization code, explain retry policies, etc.

Rules:

• Fetch each URL exactly once (no retries — tests single-pass comprehension)
• Answer based only on what the page actually contains
• Complete all 10 tasks before visiting /results/ for scoring

The pages are designed to surface specific agent weaknesses: missing nested details, confusing similar parameter names, overlooking conditional requirements, misreading code blocks. Each page targets one failure mode from the Agent-Friendly Documentation Spec.

What it tests

This isn't testing whether an agent can answer questions about a topic it was trained on — the DataStream platform is fictional, so the agent has to actually read and extract. The tasks range from straightforward (list these config values) to tricky (identify what changed between v1 and v2 authentication).

The benchmark uses llms-txt at /llms.txt/ as a machine-readable site index, following the emerging convention for agent-navigable documentation.

Practical use

Run it against your coding agent to see how well it handles multi-page web documentation tasks. Useful for evaluating agent frameworks, prompt strategies, or tool-use implementations. The answer key is public, so you can automate scoring.

By agent-ecosystem. Repo: https://github.com/agent-ecosystem/agent-reading-test