Config-drift-checker flags broken Claude Code skills

config-drift-checker, an open-source Claude Code plugin from the developer jameskomo, turns your CLAUDE.md, skills and hooks into a regression suite and runs it again each time Claude Code ships a new release. The project went up on its GitHub project page on August 31. It runs in your own GitHub Actions with your own API key, and the project says nothing leaves your repository.

The pitch fits in two lines: “Your Claude Code setup is code. Nobody tests it.” The tool tests it, and it tells you when one of those pieces stops behaving the way it used to.

From install to a red check

Setup takes two commands to install, then a slash command inside Claude Code:

• claude plugin marketplace add jameskomo/config-drift-checker
• claude plugin install config-drift-checker@jameskomo
• /config-drift-checker:setup, run from inside claude

The setup step reads what is already in the repository: CLAUDE.md, skills and hooks. From those it writes the first test cases, and the project gives three examples of what they look like: a real change to your code, a request that must not trigger a skill, and a command the guard hook must block.

Each case then runs the real agent, several times, in a throwaway copy of the repository with your plugin loaded. Every grader records a verdict and a reason, and a judge model, a second model asked to assess the transcript, writes down what it saw. The demo dashboard shows the result in practice: a pull request opened after Claude Code 2.1.248 in which one eval case regressed against the baseline, with the full report in an eval-report artifact holding every run, every grader, the tool calls and the response.

The release watch is a small job that checks npm every six hours. When a new Claude Code version appears, the suite runs against it, compares the result with your baseline and turns the check red if a score has moved. It can also post to Slack. Teams that already have cases, or only want the check in an existing pipeline, can add a single step: uses: jameskomo/config-drift-checker/action@v0.

The cases use Anthropic’s own claude plugin eval format. The author’s argument for that choice: “there is nothing of ours to learn and nothing to migrate away from later.”

Releases do change the parts these tests cover. Later, on October 16, Claude Code 2.1.301 changed hook behavior so that prompt and agent hooks run at most 12 of one event at a time while the rest wait their turn, according to the Claude Code changelog. A setup that leaned on many hooks firing together is the kind of thing a six-hourly suite would run against within hours of that release.

Hosted tiers

The plugin and the Action stay free and open. The paid hosted service is pitched at teams running the suite across many repositories: history across every repository and every Claude Code version, a chart of scores over releases, release-watch without GitHub Actions, and the same suite run against Codex and Cursor. Agent runs always bill to the customer’s own API key. “You pay us for the loop around them, never for the model,” the page says.

• Starter, $49 a month: 1 repository, 90 days of history, Claude Code only, hosted release-watch every six hours, per-run reports, GitHub login.
• Team, $149 a month: 10 repositories, 1 year of history, Claude Code, Codex and Cursor, per-run reports plus ablation (“what each skill is worth”), GitHub org access.
• Business, $399 a month: unlimited repositories and history, release-watch in under an hour, an audit export for hook enforcement, SAML and a private runner.

The with-and-without comparison that makes the demo worth reading is, on the hosted side, a report that starts only at the Team tier. The project is taking beta sign-ups now: the first teams get their suite written with them, at the Starter price.

What the setup turned out to be worth

The project published what the tool found on day one on a small Spring Boot service. The same cases ran twice: once with the team’s setup loaded and once with it removed, an ablation run that isolates what each part contributes. Each case ran three times.

• Guard hook must block git reset --hard: 1.00 with the setup, 0.33 without. The author’s reading is that without the hook the command usually runs, so “the hook is doing real work.”
• Add an endpoint across service, DTO and controller: 1.00 with, 0.50 without. The conventions skill changes how the code comes out; the verdict is “Keep it.”
• A Dockerfile request must not wake the backend skill: 1.00 in both runs. No over-triggering, nothing to fix.

The author calls the second column the interesting number. A suite that only reported the 1.00 scores would show three passing cases and say nothing about whether the hook or the skill had anything to do with it. The removed-setup run puts a figure on that: the endpoint skill changes the outcome in half the scored result, and the guard hook separates a reliably blocked destructive command from one that mostly goes through.

The report keeps each run, each grader and the judge’s reason, so “a score is never the end of the story.” The project explains why with a case from its own early testing: a hook test once passed because the model refused the command on its own, before the hook had run. “Same score, different cause.”