agent-skills-eval

A test runner for Agent Skills — the open standard from Anthropic for giving agents domain knowledge. The premise: writing a SKILL.md is easy; proving it actually makes the model better at the task is the hard part. agent-skills-eval runs each eval twice — once with_skill loaded into context, once without_skill (baseline) — has a judge model grade both outputs against the same assertions, and emits a side-by-side report.

This is the missing measurement piece for the skills ecosystem covered in mcp-vs-skills, library-skills, cli-anything, impeccable, agent-skill-linter. The other tools in that cluster handle authoring, distribution, validation, and bundling. agent-skills-eval handles the empirical question: did this skill change behavior measurably?

Listed on the watchlist — single author, four days old at ingest, 257 stars. Solid spec compliance and clean separation from any specific runtime suggest this lasts, but the agentskills.io ecosystem is itself young and the right pattern for skill evaluation hasn't been proven by reps yet.

How it works

                ┌─────────────────────────────┐
                │       same prompt           │
                └───────────────┬─────────────┘
                                │
                ┌───────────────┴─────────────┐
                ▼                             ▼
        ┌──────────────┐              ┌──────────────┐
        │ with_skill   │              │without_skill │
        │ SKILL.md in  │              │ baseline,    │
        │ context      │              │ no skill     │
        └──────┬───────┘              └──────┬───────┘
               │                             │
               ▼                             ▼
          target model                  target model
               │                             │
               ▼                             ▼
            output                        output
               │                             │
               └──────────┬──────────────────┘
                          ▼
                   ┌─────────────┐
                   │  judge      │  scores both against
                   │  model      │  the same assertions
                   └──────┬──────┘
                          ▼
                  pass / fail per side

The --baseline flag is what enables the comparison. Without it, you only get the with_skill run.

Quickstart

npx agent-skills-eval ./skills \
  --target gpt-4o-mini \
  --judge gpt-4o-mini \
  --baseline \
  --strict

Output workspace:

agent-skills-workspace/
└── iteration-1/
    ├── meta.json            # run metadata
    ├── benchmark.json       # rolled-up pass/fail per skill
    ├── eval-basic/
    │   ├── with_skill/      # output, timing, judge grading
    │   └── without_skill/   # ↑ same, with the skill stripped
    └── report/
        └── index.html       # the visual report

Skill layout

Standard agentskills.io shape. The minimum is SKILL.md plus evals/evals.json:

{
  "skill_name": "my-skill",
  "evals": [
    {
      "id": "basic",
      "prompt": "Use the attached data to summarize revenue.",
      "files": ["evals/files/input.csv"],
      "expected_output": "The response identifies the highest revenue month.",
      "assertions": ["The output identifies the highest revenue month."]
    }
  ]
}

If you skip assertions but provide expected_output, the SDK auto-promotes the expected output into a judge assertion — a minimal eval still produces meaningful pass/fail grading.

Provider model

OpenAI-compatible by default — works with OpenAI, Together, Groq, Anthropic via OpenAI-compat layers, local Llama servers. Bring any backend by implementing the Provider interface (five fields, one method) — useful for Ollama / vLLM / llama.cpp, internal APIs, mock providers in unit tests, routing layers.

SDK

For CI pipelines and custom dashboards:

import { OpenAICompatibleProvider, evaluateSkills } from "agent-skills-eval";

const provider = new OpenAICompatibleProvider({
  baseUrl: "https://api.openai.com/v1",
  apiKey: process.env.OPENAI_API_KEY!,
  model: "gpt-4o-mini",
  providerName: "openai",
});

await evaluateSkills({
  root: "./skills",
  workspace: "./agent-skills-workspace",
  baseline: true,
  concurrency: 4,
  workspaceLayout: "iteration",
  strict: true,
  target: { model: provider.model, provider },
  judge: { model: provider.model, provider },
  onEvent: consoleReporter(),
});

JSONL streaming via jsonlReporter({ file: "./events.jsonl" }).

Spec compliance

Implements the agentskills.io spec end to end:

• SKILL.md YAML frontmatter — required name and description, optional license, compatibility, metadata, allowed-tools
• Strict validation: name length, lowercase-hyphenated format, parent-directory match, description length
• Optional scripts/, references/, assets/ directories
• evals/evals.json schema: skill_name, evals[].id, prompt, expected_output, files, assertions
• Official artifact layout: iteration-N/<eval>/<mode>/outputs, timing.json, grading.json, benchmark.json
• with_skill vs without_skill baseline comparison

Beyond the spec it adds per-eval defaults, model params, tool definitions, deterministic tool_assertions, and a flat workspaceLayout for multi-skill dashboards.

What sets it apart

The with_skill / without_skill split is the load-bearing methodology choice. Most skill / prompt-eval frameworks measure absolute pass rate; this one measures lift attributable to the skill. That makes the framework usable as a regression check during skill iteration: did this rewrite of SKILL.md change anything, or am I imagining the improvement?

The judge-model approach is conventional but the framework is honest about it — the static HTML report shows the judge's reasoning per assertion, not just the score, so debugging a flaky judge is a finite problem.

Limitations

• Judge-model approach inherits the usual flakiness; deterministic tool_assertions are the escape valve when behavior is verifiable.
• TypeScript / Node only; Python skill projects need a Node sidecar to run evals.
• Costs scale at 2× (with + without each iteration); for a 50-eval skill suite, every run is 100 model calls plus 100 judge calls.
• Judge selection is a knob with real consequences; the framework doesn't guide choice — that's left to you.

Watchlist criteria

Filed under watchlist — single author, 4 days old at ingest. Re-check 2026-08-10 for: contributor count, judge-flakiness mitigation patterns, integration with the rest of the agentskills.io ecosystem, whether the agentskills.io spec stays stable enough for tools at this layer.

Cross-references

• mcp-vs-skills (skills as knowledge, MCP as capability)
• library-skills (distributed-registry distribution model)
• cli-anything (skills generated alongside CLIs)
• impeccable, agent-skill-linter (skill validation, sibling layer)
• flue (agent harness framework that consumes skills)
• features-to-steal-from-npmx (registry-design context)