Config-drift-checker flags broken Claude Code skills

The open-source config-drift-checker utility tests Claude Code configurations against every new release, with three runs per case and a baseline comparison that exposes when CLAUDE.md, skills or hooks stop behaving as intended.

According to GitHub, the tool runs inside a repository’s own GitHub Actions, uses its own API key, and keeps repository contents in place. It is designed to test the setup already in use rather than replace it.

At a glance

• Each evaluation runs in a temporary repository copy, records grader verdicts and reasons, and adds a judge model’s explanation of the observed behavior.
• A six-hour npm check detects Claude Code releases, reruns the suite against the new version, compares results with the baseline, and can post failures to Slack.
• Hosted plans begin at $49 per month, while the open-source plugin and GitHub Action remain available for teams managing their own pipeline.

Claude Code configurations increasingly carry operational rules that can be as important as application code, yet they are rarely covered by regression tests. A small change in model behavior can therefore weaken a destructive-command hook, alter a convention enforced by a skill, or create unwanted skill activation without producing an obvious failure in ordinary development. Config-drift-checker frames those rules as testable behavior, making configuration changes measurable across releases. The approach likely matters most for teams whose repositories depend on local instructions and enforcement hooks rather than on the model alone.

The first Spring Boot evaluation gave the guard hook a 1.00 score

The initial evaluation used a small Spring Boot service and tested three behaviors. With the team’s setup loaded, the guard hook blocking <code>git reset --hard</code> scored 1.00, compared with 0.33 when the setup was removed. The result shows that the hook was responsible for blocking the command in the tested cases.

The same comparison measured a conventions skill through a request to add an endpoint across the service, DTO and controller. That case scored 1.00 with the setup and 0.50 without it. A separate request for a Dockerfile scored 1.00 in both configurations, indicating that the backend skill did not over-trigger in that test.

Claude Code release checks run every six hours against a baseline

Setup requires adding the James Komo marketplace, installing the config-drift-checker plugin, and running its setup command from Claude Code. The plugin reads existing CLAUDE.md files, skills and hooks, then writes initial cases covering a real code change, a request that should not activate a skill, and a command that a guard hook must block.

Each case runs several times in a throwaway copy of the repository with the plugin loaded. Graders record both a verdict and a reason, while a judge model explains what occurred. The report preserves every run, grader, tool call and response in an eval-report artifact, so an identical score can still be traced to a different cause.

The eval format follows Anthropic’s Claude plugin specification

Existing cases can be retained, and the GitHub Action can be added to an established pipeline with a single workflow step using <code>jameskomo/config-drift-checker/action@v0</code>. The cases use Anthropic’s Claude plugin eval format, which keeps the test structure aligned with a format already used in the Claude Code ecosystem.

The project also records a failure mode from early testing: a hook case once passed because the model refused the command before the hook ran. The score did not distinguish those causes by itself, but the detailed reasons did. That distinction makes the reports useful for checking whether enforcement comes from the intended hook or from incidental model behavior.

The hosted beta starts at $49

The hosted service is aimed at teams running the suite across multiple repositories. Starter costs $49 per month for one repository and 90 days of history, Team costs $149 for unlimited repositories and one year of history, and Business costs $399 with unlimited history. All tiers include Claude Code, while Team and Business also list Codex and Cursor support.

Hosted plans add release watching without GitHub Actions, cross-repository history and score charts. Team adds ablation reporting that shows what each skill is worth, while Business adds audit export for hook enforcement. The first participating teams will have their suites written with them, and the beta is offered at the Starter price.