Claude Code 2.1.269 grades plugins with a scored eval run

Claude Code 2.1.269 ships a grader for plugin authors, and the useful half is the control run: claude plugin eval scores every test case with your plugin loaded, then runs the same case again with the plugin out of the way. The release notes on Github say results come out scored and reproducible, as JSON plus an HTML report.

At a glance

• ClaudeDevs says each case's score with and without the plugin lands in your terminal, with the full detail in an HTML report that is also published as a private artifact if your account supports it.
• In the same build, CLAUDE_CODE_WORKFLOW_MAX_CONCURRENT_AGENTS accepts a value from 1 to 256 and raises the Workflow tool's per-run limit on concurrent agents for fan-outs that are bound by inference.
• The cost and trust warning comes from ClaudeDevs: evals call the model, so they spend tokens and results vary, and a plugin's hooks and MCP servers run with your own permissions.

If you have not followed the plugin side of Claude Code: marketplaces are catalogs you add once and then install skills, agents, hooks and MCP servers from, and Anthropic's docs say Claude Code adds the official marketplace itself on the first interactive start. Measuring whether any of that helped was left to third parties. Microsoft published an eval-guide plugin that walks through a five stage evaluation lifecycle, according to its GitHub repository, and evaluation-focused subagents were already circulating in community catalogs.

Every case runs twice, and the second run has the plugin switched off

The workflow, as ClaudeDevs describes it, has four steps: write test cases, run your plugin or skill against them, score those runs, then run each case again without the plugin. The terminal shows each case's score both ways; the release notes add the JSON output and point you at claude plugin eval --help.

Think of it as a taste test with a control dish. A plugin that bundles a skill, a hook and an MCP server changes a lot about how Claude answers, and without that second run you cannot tell whether a better answer came from your plugin or from the model having a good day.

Extracted plugin archives were readable by other local users

Three fixes sit on the same fault line. A plugin archive extracted for a session could be read by other local users, extracted files kept the world-writable bits from the archive, and stale files survived a re-extraction. All three are closed in this build.

The second one is about the shell. Edit() deny rules and the write-path check did not apply to the file a Bash tee command writes, so a Bash(tee:*) allow rule reached destinations outside the working directories; it no longer covers them. The third: a deny or ask rule starting with ! applied beyond the settings source that wrote it, and now stays inside its own source, with a bare ! negation ignored.

Two smaller ones nearby: a plugin headersHelper consent prompt showed a URL path that could be misread as a different host, and plugin LSP servers that reject shutdown params, such as rust-analyzer, were left running at session end. Exit is now sent even when shutdown fails.

The Workflow tool now takes up to 256 concurrent agents per run

CLAUDE_CODE_WORKFLOW_MAX_CONCURRENT_AGENTS sets that per-run ceiling anywhere from 1 to 256, aimed at fan-outs whose bottleneck is inference rather than anything on your disk. Two more variables arrive with it.

CLAUDE_CODE_GATEWAY_MODEL_DISCOVERY_TIMEOUT_MS extends the LLM gateway's /v1/models discovery timeout, which defaults to 3 seconds, and OTEL_METRICS_INCLUDE_REPOSITORY tags OpenTelemetry metrics and events with vcs.* repository attributes; commit events pick up vcs.ref.head.* when OTEL_LOG_TOOL_DETAILS is on.

On the settings side, bashEditDiffEnabled adds a diff of the files a Bash command changed to the Bash tool result when the Bash tool handles the edits, and /output-style [name] lists and switches output styles, including over Remote Control and in cloud and other headless sessions.

Two prompt cache bugs lived on the turn after an interruption

Prompt caching only pays off when the opening of what you send matches what you sent last time; change something earlier in the conversation and the cached prefix stops matching, like a bookmark that slips when someone reprints the earlier pages.

The cache was partially invalidated on the turn after a response was cut off at the output-token limit and automatically resumed, and resuming a session after interrupting Claude mid-thought could change how earlier context was re-sent. Cloud sessions missed the cache because the first request went out before server configuration arrived; Claude Code now waits briefly for it.

Long sessions get two more repairs: transcript updates no longer re-process the whole conversation to build the collapsed tool-use summaries, and sessions could get permanently stuck on "Prompt is too long" when auto-compaction had no complete earlier exchange to summarize, mostly in Agent SDK sessions with very large prompts. After a compaction, Claude is now told the current git status instead of the one from session start.

VS Code gets an agent map and dialogs for hooks and permission rules

An "N agents" pill in the VS Code footer opens a map of the session's sub-agents, with a card per agent, a Stop agent button and read-only transcripts. Two new dialogs in the command menu list hooks and permission rules and edit them in user, project and local settings; managed, plugin and session entries stay read-only.

In cloud sessions you can take back a queued message before Claude reads it, by removing it from the queue or pressing Esc or Up, and the text returns to the message box. Free-plan users opening claude.ai/code now reach the plans page instead of a "Disabled by org admin" dead end.

In Slack, Claude Tag asks for confirmation before Connect all or Disconnect on a GitHub installation, and declines a switch to a model your organization has not enabled rather than quietly answering with a fallback.

Neither the release notes nor ClaudeDevs say how a case is scored or on what scale, and the stated limits are real ones: runs cost tokens, results vary between runs, and the plugin under test executes its hooks and MCP servers as you. In our view that is an odd pairing inside one build, which fixes plugin archives being readable by other local users and then hands you a command that asks you to only evaluate plugins you already trust.

Before a full eval run

ClaudeDevs suggests piloting with --runs 1 before committing to a full suite, and getting the command at all takes a claude update. Whether your HTML report is also published as a private artifact depends on account support, and neither source says which accounts qualify. Organization plugins enabled through managed settings now load in headless sessions and on Claude Desktop, but only once Desktop bundles this CLI version; no date is given for that.