Claude Code grades your plugin against a no-plugin run

Somewhere in the new documentation sits an error message you can do nothing about: "plugin eval is currently unavailable" means Anthropic switched the command off server-side, and nothing on your machine turns it back on. The command is claude plugin eval, described in Claude's docs for plugin and skill authors, and it runs your plugin against a suite of test cases and then runs the same cases again with the plugin removed.

At a glance

• claude plugin eval scores a Claude Code plugin against realistic prompts, and claude plugin eval init interviews you about the plugin, proposes the cases and graders, tries them and writes the files for you.
• Each case runs three times by default, then the same runs repeat with no plugin loaded, and the gap between the WITH and W/OUT scores, Δ, is what your plugin actually contributed.
• Runs, judge-scored graders and init all call the model with your credentials, so they count against your plan's usage limits or your API bill, and the printed cost is a list-price estimate.

If you have not followed the plugin layer: Claude Code launched as a research preview in February 2025, according to its release history on GitHub, and grew skills, agents, hooks and MCP servers on top of the Model Context Protocol that Anthropic published in November 2024. Most of that layer is prose. A skill's description is what decides whether Claude invokes it, and claude plugin validate only checks a plugin's files for syntax and schema errors, not its behaviour.

Each case runs three times, then three times more with no plugin

A case is a directory with a prompt and one or more graders. For each run, Claude Code starts a fresh non-interactive session with only your plugin loaded, sends the prompt, and lets Claude work until it finishes or hits the case's limits: 10 turns and 300 seconds by default, up to 200 and 3600 if you raise them.

One run of a non-deterministic agent tells you little, so each case runs three times by default and up to 50. A run scores the fraction of its graders that passed, weighted if you set weights, and the case score is the mean across its runs. A case passes when it meets --threshold, 1.0 by default, and anything below makes the command exit 1.

Then the same runs repeat with nothing loaded. That second arm is the control group, and Δ is the with-arm score minus the without-arm score. One wrinkle: a tool_used grader for the Skill tool can never pass without the plugin, so Claude Code drops it from scoring in both arms and shows it as an indicator only, rather than letting it inflate Δ.

Four of the six grader types cost nothing to run

regex, tool_used, tool_order and file_exists are computed from the transcript and the files on disk. llm and baseline call a judge model, and an llm grader passes when at least two of three judge votes say PASS. The default judge is a small fast model, and --judge-model sonnet buys a stronger one for nuanced rubrics. There are no custom-code graders.

The arithmetic of a suite: roughly cases × runs agent runs with the plugin, as many again for the baseline, plus three short judge calls per llm or baseline grader per run. Passing --ablation none drops the baseline arm and halves that cost.

Anthropic's advice on keeping scores stable is worth repeating. Grade long output with a regex over the file's contents, keep llm graders for short replies, and give each case one grader on the result and one on how Claude got there. If the Skill grader passes but Δ is negative, suspect the judge before the plugin.

A run starts in a throwaway home directory and cannot read its own cases

Each run gets a disposable home directory, working directory and Claude Code configuration, and the agent under test runs there as a claude -p child process. Your settings, hooks, CLAUDE.md files, MCP servers, other installed plugins, memory and skills are all absent, and only an allowlist of environment variables plus your own EVAL_* ones reach the run.

The agent cannot read the eval directory either, so it never sees the case's prompt definition, its graders or sibling cases. The Artifact tool is off. Anything beyond a read-only set of Read, Glob, Grep, NotebookRead, Skill, Agent, TodoWrite and the task tools is removed from the session unless you grant it with --allow-tools.

Grant Bash and every command runs under Claude Code's OS-level sandbox: writes confined to the run's workspace, your home directory and configuration unreadable, network limited to domains you grant. With no sandbox backend the run is refused rather than run unconfined, so shell-granting suites need WSL2 on Windows, and bubblewrap and socat on Linux.

Mock MCP servers are Markdown files, one per tool

Put a file at evals/mocks/<server>/<tool>.md and its body becomes what the tool returns to Claude, with {{input.field}} substitutions. A run never starts your plugin's real servers unless you ask. An expect: block guards the input, and a call that violates it aborts the run with score 0 and records why, so a case can assert what your plugin asked the server to do.

Mocks of type: agent answer with a model call and vary between runs, so completed runs save their answers under mock-recordings/ for you to copy into mocks/.replay/ and commit, which is what makes CI repeatable. In CI you pass --trust-plugin so the job never waits at the first-run trust prompt, pin both models, and gate on the exit code: 1 for a case below threshold, 2 when the --max-cost-usd ceiling stopped the run and results.json carries partial: true.

The isolation is not a security boundary. The docs say it limits what the agent under test can reach, not what the plugin's own hooks or real MCP servers can do, and a suite that passes says nothing about whether the plugin is safe. Oddly, a suite that runs into your plan's usage limit still finishes unmarked: later runs score 0, and the result reads exactly like a regression.

If the command refuses to start

Two messages mean the problem is not your suite. "plugin eval is currently in early access" means your build predates general availability, so run claude update and try again in a fresh session. "plugin eval is currently unavailable" means the server-side switch is off, and the documentation gives no timetable for its return, only the same advice to update and try later. The floor is Claude Code v2.1.269.