# agent-skills-eval (darkrishabh/agent-skills-eval)

# agent-skills-eval (darkrishabh/agent-skills-eval)

**Repo:** https://github.com/darkrishabh/agent-skills-eval
**Author:** darkrishabh
**Created:** 2026-05-06
**Stars:** 257 (as of 2026-05-10)
**License:** MIT
**Language:** TypeScript
**Docs:** https://darkrishabh.github.io/agent-skills-eval/

## What it does

A test runner for [Agent Skills](https://agentskills.io) — the open standard from Anthropic for giving agents domain knowledge. The premise: shipping a `SKILL.md` is easy; *proving* it makes the model better at the task is the hard part.

`agent-skills-eval` runs each eval **twice** — once `with_skill` loaded into context, once `without_skill` (baseline) — has a judge model grade both outputs, and emits a side-by-side report. If the skill makes no measurable difference, the report shows it. If it does, you have receipts.

It is the test framework for the Agent Skills ecosystem, separated from any specific agent runtime so it works wherever skills do.

## Architecture

```
                ┌─────────────────────────────┐
                │       same prompt           │
                └───────────────┬─────────────┘
                                │
                ┌───────────────┴─────────────┐
                ▼                             ▼
        ┌──────────────┐              ┌──────────────┐
        │ with_skill   │              │without_skill │
        │ SKILL.md in  │              │ baseline,    │
        │ context      │              │ no skill     │
        └──────┬───────┘              └──────┬───────┘
               │                             │
               ▼                             ▼
          target model                  target model
               │                             │
               ▼                             ▼
            output                        output
               │                             │
               └──────────┬──────────────────┘
                          ▼
                   ┌─────────────┐
                   │  judge      │  scores both against
                   │  model      │  the same assertions
                   └──────┬──────┘
                          ▼
                  pass / fail per side
```

The judge sees the eval's `expected_output` and `assertions` and grades each side independently. The `--baseline` flag enables the comparison; without it you only get the `with_skill` run.

## Quickstart

```bash
npx agent-skills-eval ./skills \
  --target gpt-4o-mini \
  --judge gpt-4o-mini \
  --baseline \
  --strict
```

Output workspace:

```
agent-skills-workspace/
└── iteration-1/
    ├── meta.json            # run metadata
    ├── benchmark.json       # rolled-up pass/fail per skill
    ├── eval-basic/
    │   ├── with_skill/      # output, timing, judge grading
    │   └── without_skill/   # ↑ same, with the skill stripped
    └── report/
        └── index.html       # the visual report
```

## Skill layout

Standard agentskills.io shape. Minimum is `SKILL.md`. Add `evals/evals.json` to make it evaluable.

```
my-skill/
├── SKILL.md
├── references/
├── scripts/
└── evals/
    ├── evals.json
    └── files/
        └── input.csv
```

`evals.json`:

```json
{
  "skill_name": "my-skill",
  "evals": [
    {
      "id": "basic",
      "name": "basic behavior",
      "prompt": "Use the attached data to summarize revenue.",
      "files": ["evals/files/input.csv"],
      "expected_output": "The response identifies the highest revenue month.",
      "assertions": ["The output identifies the highest revenue month."]
    }
  ]
}
```

Skip `assertions` and `expected_output` is auto-promoted to a judge assertion.

## Provider model

OpenAI-compatible by default — works with OpenAI, Together, Groq, Anthropic via OpenAI-compat layers, local Llama servers, anything that speaks the OpenAI chat API. Bring your own backend by implementing the `Provider` interface (five fields, one method).

```ts
export const provider: Provider = {
  name: "my-provider",
  model: "my-model",
  async complete(prompt: string): Promise<ProviderResult> { /* ... */ },
};
```

Useful for: local model servers (Ollama, vLLM, llama.cpp), proprietary internal APIs, mock providers in unit tests, routing layers.

## SDK

For CI pipelines, custom dashboards, multi-skill rollups:

```ts
import { OpenAICompatibleProvider, consoleReporter, evaluateSkills } from "agent-skills-eval";

const provider = new OpenAICompatibleProvider({
  baseUrl: "https://api.openai.com/v1",
  apiKey: process.env.OPENAI_API_KEY!,
  model: "gpt-4o-mini",
  providerName: "openai",
});

const result = await evaluateSkills({
  root: "./skills",
  workspace: "./agent-skills-workspace",
  baseline: true,
  concurrency: 4,
  workspaceLayout: "iteration",
  strict: true,
  target: { model: provider.model, provider },
  judge: { model: provider.model, provider },
  onEvent: consoleReporter(),
});
```

JSONL streaming via `jsonlReporter({ file: "./events.jsonl" })`.

## Reports

Static HTML built from disk artifacts. Shows pass rate by skill and by eval, assertion-by-assertion grading evidence with judge reasoning, full target output side by side for `with_skill` and `without_skill`, prompt and judge prompt details, timing and token usage, tool calls when present.

## agentskills.io compatibility

Implements the spec end to end:

- `SKILL.md` YAML frontmatter — required `name` and `description`, optional `license`, `compatibility`, `metadata`, `allowed-tools`
- Strict validation: name length, lowercase-hyphenated format, parent-directory match, description length, compatibility length
- Optional `scripts/`, `references/`, `assets/` directories
- `evals/evals.json` schema: `skill_name`, `evals[].id`, `prompt`, `expected_output`, `files`, `assertions`
- Official artifact layout: `iteration-N/<eval>/<mode>/outputs`, `timing.json`, `grading.json`, `benchmark.json`
- Baseline comparison via `with_skill` and `without_skill`

Beyond the spec: per-eval `defaults`, model `params`, tool definitions, deterministic `tool_assertions`, flat `workspaceLayout: "flat"` for multi-skill dashboards.

## YAML config

```yaml
root: ./skills
workspace: ./agent-skills-workspace
baseline: true
target: gpt-4o-mini
judge: gpt-4o-mini
baseUrl: https://api.openai.com/v1
apiKeyEnv: OPENAI_API_KEY
include: ["skills/**"]
exclude: ["**/draft-*"]
concurrency: 4
layout: iteration
strict: true
report:
  enabled: true
  title: Agent Skills Report
logging:
  format: pretty   # pretty | jsonl | silent
targetParams: { temperature: 0 }
judgeParams: { temperature: 0 }
```

CLI flags override config.

## Logging modes

`pretty` for humans, `jsonl` for machines, `silent` for quiet CI.

## Tool-call assertions

Deterministic checks for agents that call tools, not just generate text.

## Examples

`examples/basic-skill` for a complete skill folder, `examples/agent-skills-eval.yaml` for a reference config.