# agent-skills-eval

A test runner for [Agent Skills](https://agentskills.io) — the open standard from Anthropic for giving agents domain knowledge. The premise: writing a `SKILL.md` is easy; *proving* it actually makes the model better at the task is the hard part. agent-skills-eval runs each eval twice — once `with_skill` loaded into context, once `without_skill` (baseline) — has a judge model grade both outputs against the same assertions, and emits a side-by-side report.

This is the missing measurement piece for the skills ecosystem covered in [[mcp-vs-skills]], [[toolbox/library-skills]], [[toolbox/cli-anything]], [[impeccable]], [[toolbox/agent-skill-linter]]. The other tools in that cluster handle authoring, distribution, validation, and bundling. agent-skills-eval handles the empirical question: *did this skill change behavior measurably?*

Listed on the [[toolbox/watchlist|watchlist]] — single author, four days old at ingest, 257 stars. Solid spec compliance and clean separation from any specific runtime suggest this lasts, but the agentskills.io ecosystem is itself young and the right pattern for skill evaluation hasn't been proven by reps yet.

## How it works

```
                ┌─────────────────────────────┐
                │       same prompt           │
                └───────────────┬─────────────┘
                                │
                ┌───────────────┴─────────────┐
                ▼                             ▼
        ┌──────────────┐              ┌──────────────┐
        │ with_skill   │              │without_skill │
        │ SKILL.md in  │              │ baseline,    │
        │ context      │              │ no skill     │
        └──────┬───────┘              └──────┬───────┘
               │                             │
               ▼                             ▼
          target model                  target model
               │                             │
               ▼                             ▼
            output                        output
               │                             │
               └──────────┬──────────────────┘
                          ▼
                   ┌─────────────┐
                   │  judge      │  scores both against
                   │  model      │  the same assertions
                   └──────┬──────┘
                          ▼
                  pass / fail per side
```

The `--baseline` flag is what enables the comparison. Without it, you only get the `with_skill` run.

## Quickstart

```bash
npx agent-skills-eval ./skills \
  --target gpt-4o-mini \
  --judge gpt-4o-mini \
  --baseline \
  --strict
```

Output workspace:

```
agent-skills-workspace/
└── iteration-1/
    ├── meta.json            # run metadata
    ├── benchmark.json       # rolled-up pass/fail per skill
    ├── eval-basic/
    │   ├── with_skill/      # output, timing, judge grading
    │   └── without_skill/   # ↑ same, with the skill stripped
    └── report/
        └── index.html       # the visual report
```

## Skill layout

Standard agentskills.io shape. The minimum is `SKILL.md` plus `evals/evals.json`:

```json
{
  "skill_name": "my-skill",
  "evals": [
    {
      "id": "basic",
      "prompt": "Use the attached data to summarize revenue.",
      "files": ["evals/files/input.csv"],
      "expected_output": "The response identifies the highest revenue month.",
      "assertions": ["The output identifies the highest revenue month."]
    }
  ]
}
```

If you skip `assertions` but provide `expected_output`, the SDK auto-promotes the expected output into a judge assertion — a minimal eval still produces meaningful pass/fail grading.

## Provider model

OpenAI-compatible by default — works with OpenAI, Together, Groq, Anthropic via OpenAI-compat layers, local Llama servers. Bring any backend by implementing the `Provider` interface (five fields, one method) — useful for Ollama / vLLM / llama.cpp, internal APIs, mock providers in unit tests, routing layers.

## SDK

For CI pipelines and custom dashboards:

```ts
import { OpenAICompatibleProvider, evaluateSkills } from "agent-skills-eval";

const provider = new OpenAICompatibleProvider({
  baseUrl: "https://api.openai.com/v1",
  apiKey: process.env.OPENAI_API_KEY!,
  model: "gpt-4o-mini",
  providerName: "openai",
});

await evaluateSkills({
  root: "./skills",
  workspace: "./agent-skills-workspace",
  baseline: true,
  concurrency: 4,
  workspaceLayout: "iteration",
  strict: true,
  target: { model: provider.model, provider },
  judge: { model: provider.model, provider },
  onEvent: consoleReporter(),
});
```

JSONL streaming via `jsonlReporter({ file: "./events.jsonl" })`.

## Spec compliance

Implements the [agentskills.io](https://agentskills.io) spec end to end:

- `SKILL.md` YAML frontmatter — required `name` and `description`, optional `license`, `compatibility`, `metadata`, `allowed-tools`
- Strict validation: name length, lowercase-hyphenated format, parent-directory match, description length
- Optional `scripts/`, `references/`, `assets/` directories
- `evals/evals.json` schema: `skill_name`, `evals[].id`, `prompt`, `expected_output`, `files`, `assertions`
- Official artifact layout: `iteration-N/<eval>/<mode>/outputs`, `timing.json`, `grading.json`, `benchmark.json`
- `with_skill` vs `without_skill` baseline comparison

Beyond the spec it adds per-eval `defaults`, model `params`, tool definitions, deterministic `tool_assertions`, and a flat `workspaceLayout` for multi-skill dashboards.

## What sets it apart

The `with_skill` / `without_skill` split is the load-bearing methodology choice. Most skill / prompt-eval frameworks measure absolute pass rate; this one measures *lift attributable to the skill*. That makes the framework usable as a regression check during skill iteration: did this rewrite of `SKILL.md` change anything, or am I imagining the improvement?

The judge-model approach is conventional but the framework is honest about it — the static HTML report shows the judge's reasoning per assertion, not just the score, so debugging a flaky judge is a finite problem.

## Limitations

- Judge-model approach inherits the usual flakiness; deterministic `tool_assertions` are the escape valve when behavior is verifiable.
- TypeScript / Node only; Python skill projects need a Node sidecar to run evals.
- Costs scale at 2× (with + without each iteration); for a 50-eval skill suite, every run is 100 model calls plus 100 judge calls.
- Judge selection is a knob with real consequences; the framework doesn't guide choice — that's left to you.

## Watchlist criteria

Filed under [[toolbox/watchlist]] — single author, 4 days old at ingest. Re-check 2026-08-10 for: contributor count, judge-flakiness mitigation patterns, integration with the rest of the agentskills.io ecosystem, whether the agentskills.io spec stays stable enough for tools at this layer.

## Cross-references

- [[mcp-vs-skills]] (skills as knowledge, MCP as capability)
- [[toolbox/library-skills]] (distributed-registry distribution model)
- [[toolbox/cli-anything]] (skills generated alongside CLIs)
- [[impeccable]], [[toolbox/agent-skill-linter]] (skill validation, sibling layer)
- [[toolbox/flue]] (agent harness framework that consumes skills)
- [[features-to-steal-from-npmx]] (registry-design context)