How to test Claude Code plugins with evals

Plugin authors often ship on intuition: the skill fired once, the reply looked right. claude plugin eval replaces that with a measured comparison. It runs your plugin against a suite of prompts, scores each result with graders, and — by default — repeats the same cases with no plugin loaded. The difference, Δ, is what the plugin actually contributed.

This walkthrough follows the official Claude Code flow: check requirements, init a suite, run with versus without the plugin, read WITH / W/OUT / Δ, then gate changes in CI.

Check requirements first

You need Claude Code v2.1.269 or later. Run claude --version to check and claude update to upgrade. You also need a plugin directory with a plugin.json or .claude-plugin/plugin.json manifest, or a skills-directory plugin.

Eval runs, judge-scored graders, and claude plugin eval init all call the model with the same credentials as a normal session, so they count against your plan’s usage limits or your API bill. When the command reports a cost, that figure is a list-price estimate of those calls.

Use evals for behavior. To check a plugin’s files for syntax and schema errors instead, use claude plugin validate.

How a run is scored

An eval suite lives under evals/ inside your plugin. Each case is a subdirectory with a prompt plus one or more graders: a regex over the reply, whether a tool was called, whether a file exists, or a rubric a second model judges.

For each run, Claude Code starts a fresh, isolated non-interactive session with only your plugin loaded, sends the prompt, and lets Claude work until it finishes or hits the case’s turn or time limit. Graders then check the final reply, the transcript, or a file Claude created.

Each case runs three times by default. A run’s score is the fraction of its graders that passed, weighted if you set weights, and the case’s score is the mean across its runs. A case passes when its score meets --threshold, 1.0 by default.

A high score does not prove the plugin helped. By default the same runs repeat with no plugin loaded. You get WITH and W/OUT; their difference, Δ, is what the plugin contributed. If a case scores 1.0 both with and without the plugin, the plugin isn’t what made it pass.

Init a suite

From the plugin root — the directory that contains plugin.json or .claude-plugin/plugin.json — run:

claude plugin eval init

If Claude Code doesn’t already trust this directory it first asks Trust this plugin directory?; answer y. An interactive session then opens. Claude reads your plugin, asks what a good result looks like, proposes prompts that should and shouldn’t trigger the plugin, designs graders, pilots them once, and writes one case directory per prompt under evals/. When the suite is ready, exit with /exit or Ctrl+D.

If you already have a session open at the plugin root, ask Claude there to run claude plugin eval init. To write a blank case yourself (no run):

claude plugin eval init --bare first-case

If evals/ is already taken, add "experimental": { "evals": "quality/evals" } in plugin.json, or pass --eval-dir quality/evals to both claude plugin eval and claude plugin eval init. Use a relative path of plain directory names; an absolute path or one containing .. isn’t accepted.

Run the suite and read Δ

claude plugin eval .

Each case runs three times with your plugin and three times without it, so one case is six runs. When the suite finishes you see a summary like this:

CASE        WITH  W/OUT Δ      RUNS COST    NOTES
first-case  1.00  0.33  +0.67  6    $0.41

1 case(s) · mean Δ +0.67 · 74s · $0.41
Report: /Users/you/my-plugin/evals/results/2026-09-10T17-02-11-482Z/report.html
Published: https://claude.ai/... · keep local next time with --no-publish

WITH is the score with your plugin loaded, W/OUT is the score without it, and a positive Δ means the plugin raised the score. COST is a list-price estimate. NOTES shows the highest-weight failing grader’s explanation, or the run’s error, from the with-arm.

Open the Published: URL, or the Report: path when no Published: line appears, to see each grader’s verdict for every run. A common first finding is a Δ near zero with the case’s tool_used: Skill grader failing: Claude isn’t choosing your skill on natural phrasing. Adjust the skill’s description and run claude plugin eval . again.

To iterate on one case cheaply (noisy until you restore the default three runs):

claude plugin eval . --case <case-name> --runs 1 --ablation none

Replace <case-name> with a directory name under evals/. --ablation none runs only the with-arm and halves the cost.

Graders, tools, and trust

claude plugin eval init writes cases as prompt.md plus files under graders/, with optional case.yaml for context fields. Set max_turns, timeout_seconds, model, tags, and allowed_tools in prompt.md frontmatter.

Of the six grader types, regex, tool_used, tool_order, and file_exists are computed from the transcript and files and cost nothing. llm and baseline call a judge model. The judge is a small fast model by default; pass --judge-model sonnet or a full model ID for nuanced rubrics. Keep llm graders for short outputs with concrete PASS and FAIL conditions; grade long files with regex.

In a two-arm run, every tool_used grader whose tool is Skill, and any grader you mark arm: with-only, is excluded from the score in both arms so it doesn’t inflate Δ. Set arm: both to score a grader in both arms — what you want for a “must not invoke the skill” check.

Runs never stop to ask for permission. Built-in tools you didn’t grant — Bash, Write, Edit, WebFetch, and WebSearch — are removed from the session. Grant extras with:

claude plugin eval . --allow-tools Write Edit "Bash(npm test *)"

When you grant Bash in any form, every command runs under Claude Code’s OS-level sandbox. If you grant Bash or PowerShell on a machine with no sandbox backend, Claude Code refuses each run rather than running it unconfined. Native Windows has no backend, so run shell-granting suites under WSL2; on Linux, install bubblewrap and socat first.

The first run asks Trust this plugin directory?. --trust-plugin skips that prompt; pass it only for a plugin whose code and suite you’d run yourself. A case’s scaffold_script runs as you, outside the agent’s sandbox, and only when you pass --scaffold, so use that flag only for suites you or your organization wrote.

Gate plugin changes in CI

Write JSON for archiving, fail the job on the exit code, skip the trust prompt, pin both models so a model rollout isn’t mistaken for a plugin regression, keep the report local, and set a cost ceiling:

claude plugin eval . \
  --trust-plugin \
  --json results.json \
  --threshold 0.8 \
  --model claude-sonnet-5 \
  --judge-model claude-haiku-4-5 \
  --no-publish \
  --max-cost-usd 20

Exit codes:

• 0 — every case scored at or above --threshold and every case file loaded
• 1 — a case scored below the threshold, a case file failed to load, no cases were found, a run couldn’t be started, the plugin directory isn’t trusted and --trust-plugin wasn’t passed, or an option was invalid
• 2 — partial run: the --max-cost-usd ceiling was hit, or your credential was rejected. results.json is still written with partial: true
• 130 — interrupted; partial results are written
• 143 — terminated, such as by a CI timeout

--max-cost-usd is a ceiling on the list-price estimate, not on plan usage. It is checked before each run starts. Runs already in flight finish, so spend can pass the ceiling by those runs.

From your plugin root, run claude plugin eval init, then claude plugin eval .. Treat a near-zero Δ as a signal to tighten skill descriptions or graders — not as a pass.

Sources

• Plugin evals
• Plugins reference