Claude Code grades your plugin against a no-plugin run

Anthropic on September 11 added a test harness for plugins to Claude Code. claude plugin eval runs a plugin against a suite of realistic prompts, scores what Claude did, and then reruns the same prompts with the plugin removed, so authors can see what the plugin itself contributed. The Claude Code documentation is written for plugin and skill authors who already have a working plugin, and for teams that want to gate plugin changes in CI.

Each case is a directory under evals/ in the plugin. It holds a prompt phrased the way a user would type it, ideally without naming the skill, plus one or more graders, each a pass/fail check on the result. You don’t have to write the suite by hand: claude plugin eval init asks about your plugin, proposes cases and graders, tries them and writes the files. You need Claude Code v2.1.269 or later. Eval runs, judge-scored graders and init itself all call the model with your credentials, so they count against your plan’s usage limits or your API bill. Any cost the command reports is a list-price estimate.

What the baseline arm measures

A high score alone proves little, because Claude might have passed without the plugin. So by default every case runs in two arms. In the with-arm the plugin is loaded. The without-arm repeats the same number of runs with no plugin at all. The report shows both scores, WITH and W/OUT, and their difference, Δ. A case that scores 1.0 in both arms passed for reasons that have nothing to do with the plugin.

Anthropic made a similar argument when it added evals to its skill-creator plugin, in an earlier post on skill testing. If the base model starts passing a skill’s evals without the skill loaded, the company said, the skill’s techniques may have been folded into the model’s default behavior. The skill isn’t broken. It just isn’t needed anymore. The new command keeps its own case format, separate from the evals/evals.json file skill-creator uses.

Comparing the arms needs one correction. A check like “the skill was invoked” can never pass without the plugin, so counting it would drag the without-arm toward zero and inflate Δ. Claude Code therefore leaves such graders out of the score in both arms and shows them in the with-arm as indicators only, marked scored: false. That covers every tool_used grader whose tool is Skill, and any grader you mark arm: with-only. Setting arm: both forces one back into scoring, which is what a “must not invoke the skill” check with min: 0 and max: 0 needs. A side effect: under --ablation none, which runs only the with-arm, nothing is excluded, so the same suite can produce a different absolute score in the two modes.

Each case runs three times by default, since one run of a non-deterministic agent tells you little. A run’s score is the share of its graders that passed, weighted if you set weights, and the case score is the mean across runs. In model calls, a suite makes roughly cases × runs agent runs with the plugin, as many again for the baseline, plus three short judge calls per llm or baseline grader per run. Running only the with-arm halves the cost, and the docs suggest doing so while you are still iterating on graders.

Six graders, no custom code

There are six grader types. Four of them, regex, tool_used, tool_order and file_exists, are computed from the transcript and files and cost nothing. The other two call a judge model: llm grades the run against a rubric, and baseline asks whether the run satisfies the criteria at least as well as a reference transcript. An llm grader passes when the judge votes PASS in at least two of three votes. The judge is a small fast model by default, and --judge-model sonnet or a full model ID swaps in a stronger one. There are no custom-code graders.

The docs warn that a judge’s verdict can differ between runs, and differs more the longer the text it has to read. A judge looking at the session trace only sees its first 12 and last 12 messages. For long output such as a generated file, the advice is to use a regex over the file’s contents, which checks the whole file the same way every time, and to keep rubric graders for short outputs written as concrete PASS and FAIL conditions. Each case should get one grader on the result and one on how Claude got there, such as tool_used or tool_order. When a case’s tool_used: Skill check passes but Δ is negative, the docs say to suspect the judge before the plugin: a small judge can fail a correct answer only because it is formatted differently from what the rubric describes.

The opposite pattern carries its own diagnosis. If the plugin loaded, Δ sits near zero and the skill-invoked check fails, that’s usually a real finding: the skill’s description doesn’t trigger on how the prompt is phrased. The fix is to adjust the description and rerun the same suite.

How sealed is a run?

Each run is a fresh, non-interactive claude -p child process with a throwaway home directory, working directory and configuration, and only the plugin under test loaded. Your user settings, hooks, CLAUDE.md files, MCP servers, other plugins, memory and skills are all absent, and most of your shell environment is withheld; extra variables have to be passed as EVAL_*. The agent also can’t read the eval directory, so it never sees the prompt file, the graders or sibling cases. Managed policy an administrator deployed still applies, so results on a managed machine can differ.

Runs never stop to ask for permission. Built-in tools that need a grant you didn’t give, including Bash, Write, Edit, WebFetch and WebSearch, are removed from the session, and you grant them with --allow-tools. Granted Bash runs under Claude Code’s OS-level sandbox, with writes confined to the run’s workspace and network limited to domains you grant. On a machine with no sandbox backend, Claude Code refuses the run outright instead of running it unconfined. Native Windows has no backend, so shell-granting suites need WSL2, and Linux needs bubblewrap and socat installed.

MCP-dependent plugins get mocks. One Markdown file per tool under evals/mocks/<server>/ supplies the tool’s response, and real servers start only if you pass --allow-real-servers or --mocks off. An expect: block in a mock guards the input Claude sends: a violating call aborts the run with score 0 and records why, so a case can assert what the plugin asked the server to do. A type: agent mock has a small model play the server, and its answers are saved so you can commit them as replays and make CI runs repeatable.

The isolation limits the agent, not the plugin’s own code. Evaluating a plugin is the same trust decision as loading it with --plugin-dir, and hooks and real MCP servers run outside the sandbox where they could touch the files the graders read. In the docs’ words, “a suite that passes says nothing about whether the plugin is safe.”

Running it in CI

For CI, the docs recommend --json for an archivable result, --trust-plugin so the job never stalls at the first-run trust prompt, both models pinned “so a model rollout isn’t mistaken for a plugin regression”, a local-only report and a --max-cost-usd ceiling. That ceiling applies to the list-price estimate, not plan usage, and runs already in flight finish, so spend can overshoot it. Exit code 0 means every case met the threshold. Code 1 covers a case below it, along with load failures and refused runs. Code 2 marks a partial run after the cost ceiling or a rejected credential, with partial: true in the JSON.

The default --threshold is 1.0, so the command exits 1 when any case scores below perfect, a default strict enough that the troubleshooting section lists “the run exits 1 but the results look fine” among the common problems.

One trap shows up only in the data. If the account hits its usage limit or an API rate limit partway through a suite, every later run ends with that error, gets graded on whatever it produced, and usually scores 0. The suite still finishes and isn’t marked partial, so the result can look like a regression. The limit message turns up in the NOTES column or in cases[].arms.with[].error in the JSON.

Every run writes report.html, a single self-contained file that makes no external requests, so it can be attached to a CI job or opened from disk. Users signed in with a claude.ai subscription who have artifacts available also get it published as a private artifact, unless they pass --no-publish; with API-key authentication the local file is the report.

Availability is still controlled from Anthropic’s side. A build that predates general availability prints “plugin eval is currently in early access” and needs claude update, while “plugin eval is currently unavailable” means Anthropic has switched the command off server-side, and nothing on your machine turns it back on.