agent-skills-eval (darkrishabh/agent-skills-eval)

Repo: https://github.com/darkrishabh/agent-skills-eval Author: darkrishabh Created: 2026-05-06 Stars: 257 (as of 2026-05-10) License: MIT Language: TypeScript Docs: https://darkrishabh.github.io/agent-skills-eval/

What it does

A test runner for Agent Skills — the open standard from Anthropic for giving agents domain knowledge. The premise: shipping a SKILL.md is easy; proving it makes the model better at the task is the hard part.

agent-skills-eval runs each eval twice — once with_skill loaded into context, once without_skill (baseline) — has a judge model grade both outputs, and emits a side-by-side report. If the skill makes no measurable difference, the report shows it. If it does, you have receipts.

It is the test framework for the Agent Skills ecosystem, separated from any specific agent runtime so it works wherever skills do.

Architecture

                ┌─────────────────────────────┐
                │       same prompt           │
                └───────────────┬─────────────┘
                                │
                ┌───────────────┴─────────────┐
                ▼                             ▼
        ┌──────────────┐              ┌──────────────┐
        │ with_skill   │              │without_skill │
        │ SKILL.md in  │              │ baseline,    │
        │ context      │              │ no skill     │
        └──────┬───────┘              └──────┬───────┘
               │                             │
               ▼                             ▼
          target model                  target model
               │                             │
               ▼                             ▼
            output                        output
               │                             │
               └──────────┬──────────────────┘
                          ▼
                   ┌─────────────┐
                   │  judge      │  scores both against
                   │  model      │  the same assertions
                   └──────┬──────┘
                          ▼
                  pass / fail per side

The judge sees the eval's expected_output and assertions and grades each side independently. The --baseline flag enables the comparison; without it you only get the with_skill run.

Quickstart

npx agent-skills-eval ./skills \
  --target gpt-4o-mini \
  --judge gpt-4o-mini \
  --baseline \
  --strict

Output workspace:

agent-skills-workspace/
└── iteration-1/
    ├── meta.json            # run metadata
    ├── benchmark.json       # rolled-up pass/fail per skill
    ├── eval-basic/
    │   ├── with_skill/      # output, timing, judge grading
    │   └── without_skill/   # ↑ same, with the skill stripped
    └── report/
        └── index.html       # the visual report

Skill layout

Standard agentskills.io shape. Minimum is SKILL.md. Add evals/evals.json to make it evaluable.

my-skill/
├── SKILL.md
├── references/
├── scripts/
└── evals/
    ├── evals.json
    └── files/
        └── input.csv

evals.json:

{
  "skill_name": "my-skill",
  "evals": [
    {
      "id": "basic",
      "name": "basic behavior",
      "prompt": "Use the attached data to summarize revenue.",
      "files": ["evals/files/input.csv"],
      "expected_output": "The response identifies the highest revenue month.",
      "assertions": ["The output identifies the highest revenue month."]
    }
  ]
}

Skip assertions and expected_output is auto-promoted to a judge assertion.

Provider model

OpenAI-compatible by default — works with OpenAI, Together, Groq, Anthropic via OpenAI-compat layers, local Llama servers, anything that speaks the OpenAI chat API. Bring your own backend by implementing the Provider interface (five fields, one method).

export const provider: Provider = {
  name: "my-provider",
  model: "my-model",
  async complete(prompt: string): Promise<ProviderResult> { /* ... */ },
};

Useful for: local model servers (Ollama, vLLM, llama.cpp), proprietary internal APIs, mock providers in unit tests, routing layers.

SDK

For CI pipelines, custom dashboards, multi-skill rollups:

import { OpenAICompatibleProvider, consoleReporter, evaluateSkills } from "agent-skills-eval";

const provider = new OpenAICompatibleProvider({
  baseUrl: "https://api.openai.com/v1",
  apiKey: process.env.OPENAI_API_KEY!,
  model: "gpt-4o-mini",
  providerName: "openai",
});

const result = await evaluateSkills({
  root: "./skills",
  workspace: "./agent-skills-workspace",
  baseline: true,
  concurrency: 4,
  workspaceLayout: "iteration",
  strict: true,
  target: { model: provider.model, provider },
  judge: { model: provider.model, provider },
  onEvent: consoleReporter(),
});

JSONL streaming via jsonlReporter({ file: "./events.jsonl" }).

Reports

Static HTML built from disk artifacts. Shows pass rate by skill and by eval, assertion-by-assertion grading evidence with judge reasoning, full target output side by side for with_skill and without_skill, prompt and judge prompt details, timing and token usage, tool calls when present.

agentskills.io compatibility

Implements the spec end to end:

SKILL.md YAML frontmatter — required name and description, optional license, compatibility, metadata, allowed-tools
Strict validation: name length, lowercase-hyphenated format, parent-directory match, description length, compatibility length
Optional scripts/, references/, assets/ directories
evals/evals.json schema: skill_name, evals[].id, prompt, expected_output, files, assertions
Official artifact layout: iteration-N/<eval>/<mode>/outputs, timing.json, grading.json, benchmark.json
Baseline comparison via with_skill and without_skill

Beyond the spec: per-eval defaults, model params, tool definitions, deterministic tool_assertions, flat workspaceLayout: "flat" for multi-skill dashboards.

YAML config

root: ./skills
workspace: ./agent-skills-workspace
baseline: true
target: gpt-4o-mini
judge: gpt-4o-mini
baseUrl: https://api.openai.com/v1
apiKeyEnv: OPENAI_API_KEY
include: ["skills/**"]
exclude: ["**/draft-*"]
concurrency: 4
layout: iteration
strict: true
report:
  enabled: true
  title: Agent Skills Report
logging:
  format: pretty   # pretty | jsonl | silent
targetParams: { temperature: 0 }
judgeParams: { temperature: 0 }

CLI flags override config.

Logging modes

pretty for humans, jsonl for machines, silent for quiet CI.

Tool-call assertions

Deterministic checks for agents that call tools, not just generate text.

Examples

examples/basic-skill for a complete skill folder, examples/agent-skills-eval.yaml for a reference config.