Poka-Yoke skills cost Claude Code its eye for SQL bugs

Poka-Yoke, a mistake-proofing skill pack for Claude Code published on Github, raises the rate at which an agent names what its own fix makes impossible from 42% to 81%. The same pack cuts the rate at which agents flag a raw SQL interpolation as an injection vector from 92% to 69%.

At a glance

• The pack bundles 11 skills, a dependency-free hazard scanner for TypeScript, Python, Go, Rust and SQL, and installable pre-commit, CI, lint and hook devices, all built on Shigeo Shingo's factory-floor method.
• Across 591 blind-graded runs, Fable 5 gained 8.3 points over its no-skill baseline, Opus 5 3.6, Sonnet 5 8.6 and Haiku 4.5 12.9, with all four 95% intervals excluding zero.
• Poka-Yoke does not auto-trigger: ten realistic queries put to fresh agents with the plugin installed produced no skill invocation, so the modes run from slash commands such as /poka-yoke:audit.

The pack's own framing is unusually blunt about the exchange it demands: loading a method changes what a model attends to, and attention is finite. That reads as rare accounting in a skills ecosystem where most releases publish capability claims and no costs. The measured benefit also tracks available headroom rather than model quality, which likely limits the payoff on frontier models that already reach for most of these devices unprompted.

Every Claude model beat its no-skill baseline by 3.6 to 12.9 points

Fable 5 went from 88.7% to 97.0%, Opus 5 from 92.7% to 96.4%, Sonnet 5 from 79.8% to 88.5% and Haiku 4.5 from 58.2% to 71.1%, across thirteen scenarios run under two configurations. The Claude matrix alone accounts for 445 of the 591 runs.

Of the 52 scenario-by-model cells, 30 improved, 13 were unchanged and nine regressed, for a mean of plus 8.3 points. Chance alone would produce roughly 18 negative cells, with a simulated 95% range of 12 to 25, so the author reads the scarcity of regressions as the signal.

The largest movements sit on the weakest model and the build-oriented scenarios: ops on Haiku 4.5 went from 29% to 92%, build-endpoint on Fable 5 from 61% to 100%, and agent-guardrails on Sonnet 5 from 59% to 91%. Those are pass rates against pre-written assertions.

SQL injection detection falls from 92% to 69% with the skills loaded

Measured over the same 591 runs, the pack takes proposing a concrete device per finding from 62% to 100%, and noting that pre-commit is bypassable and must be backed by CI from 35% to 92%. Naming what a design forecloses moves from 42% to 81%.

Two behaviours move the other way. Identifying a raw SQL interpolation as an injection vector drops from 92% to 69%, and explaining why a silently wrong number beats a failed pipeline drops from 54% to 31%. The author frames that trade as the product rather than a caveat about it.

The guidance is explicit that a comment, a docstring or a line in CLAUDE.md is not a poka-yoke but training, which degrades, while a device does not. Findings are classified on a four-rung ladder, from a mistake made impossible down to documentation, which scores as no device at all.

The detector ships 42 pattern rules, 19 of them on by default

The plugin is MIT-licensed and installs with /plugin marketplace add rainmanjam/poka-yoke. Its scanner needs Python 3.9+ and the standard library only, runs without the plugin installed, and covers TypeScript, Python, Go, Rust and SQL. It reports 20 of the 28 catalogued hazard shapes, the ones detectable in text, while the remaining eight are guidance only.

Of the 42 pattern rules, 19 are enabled by default, because 23 are covered better by a real linter and the detector names the linter instead. It flags swallowed errors, unbounded deletes, durations with no unit, money as a float and retryable effects with no idempotency key.

The skills load on 19 agent runtimes, 10 of them with a native manifest. Through the same harness and blind grader, Codex went from 74.2% to 91.0% and agy from 64.6% to 78.1%, though the audit skill cannot run on agy at all, since plan mode refuses to execute the bundled detector.

The control arm that has not run

Grading is done by Haiku 4.5 rather than a human, against assertions the pack's author wrote before the runs. The baseline is no skill rather than a competing methodology, and a control arm is designed but unrun, so whether any structured method in context would perform similarly is untested. Every run is the first turn of a fresh session, which measures the ceiling rather than what survives accumulated context.