Skip to content

Claude Code

Poka-Yoke skills cost Claude Code its eye for SQL bugs

10 min read

Written by an AI model from the linked sources and checked against them by a second model. How we work

All about: Claude Code versions and changes →
Image: github.com

A developer who goes by rainmanjam posted Poka-Yoke to Show HN on August 25. It is a Claude Code plugin with 11 skills, a hazard scanner with no dependencies, and a benchmark. The most useful table in that benchmark shows what the skills make worse.

Most of the pack comes down to one habit, as the project’s README on GitHub puts it: the agent should say what its fix makes impossible, not just what to fix. With no skill loaded, models close a design by naming what it rules out 42% of the time. With the skills loaded, they do it 81% of the time. The README calls this “the difference between advice you agree with and a constraint you can rely on.” The plugin is MIT-licensed and installs with /plugin marketplace add rainmanjam/poka-yoke, then /plugin install poka-yoke@poka-yoke.

A ladder and three lenses

Poka-yoke means mistake-proofing. Shigeo Shingo formalized it within the Toyota Production System in the 1960s, and he built it on the idea that a mistake should not be allowed to become a defect. The README uses his best-known example. Assemblers kept forgetting a spring inside a switch, so he had them lay both springs in a dish first. A spring left in the dish showed the error before the unit moved on. The plugin says it uses this taxonomy as a working tool, not a metaphor. Every finding gets classified on two axes, and the author argues that this classification is what stops the output from turning into generic code review.

The first axis asks what happens when the mistake occurs. It is a strict ladder, and the skills are told to climb as high as they can afford. Rung 1 is Control: the mistake is impossible. Examples are a type that won’t compile, a NOT NULL or CHECK constraint, a required argument, a PreToolUse deny, or branch protection. Rung 2 is Warning: the mistake can happen, but it is announced as it happens, through a lint error, a failing CI gate, a runtime assertion, or a confirmation that names the exact object. Rung 3 is Detection: the mistake ships and tests, monitoring or reconciliation find it later. Docs, comments, training and “be careful” sit at rung 0, which doesn’t count as poka-yoke at all.

The second axis uses Shingo’s three detection methods as lenses for reading code. Contact asks whether the wrong thing can fit. The fixes are distinct types, branded IDs (nominal wrappers that stop one string ID standing in for another) and units in the type. Fixed-value asks whether a wrong count or an incomplete set can get through, which leads to exhaustive matches, row-count guards and config validated at boot. Motion-step asks whether steps can happen out of order, and its answers are typestate, state machines and idempotency keys.

The README singles out one line as doing most of the work:

A comment, a docstring, a wiki page, a review checklist, or a line in CLAUDE.md saying “don’t do X” is not a poka-yoke. It is training, and training degrades. A device does not.

An audit finding shows the format. “Account IDs can be swapped in transfer()” at src/payments/transfer.ts:42 is filed as money movement, silent failure. The mistake is calling transfer(dst, src) with the accounts reversed, which “compiles, passes review, silent at runtime.” The device is to brand AccountId as SourceAccount and DestinationAccount, which puts it on the Control rung. The skills split by domain: design, audit, retro, ops, authz, ux, data, llm, guardrails and agent-guardrails, plus a general poka-yoke skill that routes to the others. Only the mode you call gets loaded.

What the skills make worse

The author ran the same 591 blind-graded runs for everything below. The skills make responses more constructive and a bit worse at spotting the specific defect already in front of the model:

  • Proposes a concrete device per finding, not “add validation”: 62% without the skill, 100% with it
  • Names what the design makes impossible: 42% → 81%
  • Notes that pre-commit can be bypassed and must be backed by CI: 35% → 92%
  • Identifies a raw SQL interpolation as an injection vector: 92% → 69%
  • Explains why a silently wrong number is worse than a failed pipeline: 54% → 31%

The README’s explanation is that loading a method changes what a model pays attention to, and attention is finite. Its advice follows from that. If you want the bug in front of you found, use a reviewer. If you want the code reshaped so that whole class of bug can’t be written, use this. “That trade is the product, not a caveat about it,” the author writes.

How much it helps, and whom

The Claude matrix has 445 runs: thirteen scenarios, four models, two configurations. A Haiku 4.5 grader scored each response against assertions written before the runs, without seeing which configuration produced it. Nine of the scenarios start with a user who has already applied or proposed a fix that isn’t enough, so agreeing with them scores badly. The other four are greenfield builds. Every model beat its no-skill baseline, and all four 95% intervals exclude zero:

  • Fable 5: 88.7% → 97.0%, +8.3 pp, interval [+1.1, +15.5], 64s → 91s per response
  • Opus 5: 92.7% → 96.4%, +3.6 pp, [+0.2, +7.0], 130s → 172s
  • Sonnet 5: 79.8% → 88.5%, +8.6 pp, [+2.2, +15.1], 88s → 129s
  • Haiku 4.5: 58.2% → 71.1%, +12.9 pp, [+0.9, +24.9], 40s → 60s

So on Opus 5, every response took about 40 seconds longer for a gain of only 3.6 points, and that gain has a lower bound just above zero. The author is upfront that the no-skill baseline is already strong. Without any help, current models already reach for row-level security with FORCE, expand/contract migrations and hooks over prose. The README calls the result “an improvement to something already competent rather than a missing capability.”

Across the 52 scenario-by-model cells, a cell’s baseline correlates with its gain at r = −0.52. Cells starting below 50% gain 19 points on average, and cells starting above 95% lose 0.9. On average the skills close 36% of the remaining headroom. The biggest jumps are in scenarios about building devices: ops on Haiku 4.5 goes from 29% to 92%, and build-endpoint on Fable 5 from 61% to 100%. Thirty cells improved, 13 didn’t change and 9 regressed. The author estimates that chance alone would produce about 18 regressions, with a simulated 95% range of 12 to 25, and reads the low count as a sign the effect is consistently positive.

The exception is Haiku 4.5 on build tasks. It drops 8.6 points there while taking 2.12 times as long, the only clearly negative result in the suite. On the refund-endpoint task, Opus 5 goes from 83.3% to 100% and Haiku 4.5 falls from 44.4% to 33.3%. The README’s explanation is that the smallest model “spends its budget on the method and ships less of the thing.” It says that pairing is the one configuration it doesn’t recommend. Haiku’s results also became less consistent: the spread across scenarios grew from 16.7 to 24.1, while Fable 5’s narrowed from 11.8 to 4.0. Longer answers don’t explain the gains. Across all 52 cells, how much longer an answer got correlates with how much better it scored at only r = 0.30.

Two runtimes outside Claude Code went through the same harness and grader. OpenAI Codex, running gpt-5.6-terra in a read-only sandbox, went from 74.2% to 91.0%. agy, running Gemini 3.1 Pro in plan mode, went from 64.6% to 78.1% on 12 of the 13 scenarios. Neither number is folded into the headline mean. A Codex reply runs 50–200 words against 400–800 for Claude, so only the change within each runtime is a fair comparison. The audit skill can’t run on agy at all, because plan mode refuses to execute the bundled detector. The other manifests, out of 19 runtimes in total, are checked structurally in CI but not behaviourally.

The write-up also records its own measurement mistakes. An authz result on Sonnet 5 was first reported as a 16-point regression. It turned out the aggregation was reading 264 of 486 runs, because a loop defaulted to three runs per cell after a seven-run sweep. An assertion was also failing responses for where they put the SQL injection finding, not whether they caught it. A fix for Haiku in agent-guardrails measured 88% when it landed, but the committed runs score 33%, below the 38% it started from. The author kept the earlier figure in the table on purpose.

The author lists what the benchmark doesn’t show. Blind grading controls bias, not accuracy. The assertions came from the skill’s author. Every run is the first turn of a fresh session, which measures the ceiling, not what survives an afternoon of built-up context. The baseline is no skill at all. The README says the comparison a team actually faces is against the CLAUDE.md or AGENTS.md rules file it has already written, and that comparison “has not been run yet.” A control arm is designed but hasn’t been run.

It doesn’t trigger on its own

Poka-Yoke runs when you call it, with /poka-yoke:audit, /poka-yoke:design and the other modes, or when you ask for it by name (“poka-yoke this repo”). The author tested whether Claude would reach for it unprompted. Ten realistic queries went to fresh agents with the plugin installed and no hint that it existed, including tenant isolation, workspace-deletion UX and a Friday migration that drops a column. None of the agents invoked a poka-yoke skill. One reasoned about skills explicitly and chose hookify instead. Three rewrites with the skill-creator description optimizer changed nothing, and the README points to an open claude-code issue that collects similar reports.

The plugin ships a middle option, assets/devices/claude-hooks/suggest_poka_yoke.py, a UserPromptSubmit hook. It matches the prompt against each mode’s vocabulary and injects an instruction naming that skill. The author puts it on the Warning rung, since an injected instruction is still just an instruction. For anything important, the README concludes, “the slash command is the device and everything else is a convenience.”

The scanner, without an agent

The detector needs only Python 3.9+ and the standard library, so there is no dependency supply chain. It covers TypeScript, Python, Go, Rust and SQL. The hazard catalog lists 28 shapes. The scanner reports the 20 that can be spotted in text, through 42 pattern rules plus two AST checks. Nineteen rules are on by default, and for the other 23 it names a real linter that does the job better (--all runs them anyway). It flags adjacent same-type parameters, swallowed errors, unbounded deletes, durations with no unit, money stored as a float, and retryable effects with no idempotency key. Each finding is tagged with its catalog ID and the device that would close it. The author warns to expect real false positives from the pattern rules.

You can try it on a repository without installing the plugin. Clone it with git clone --depth 1 https://github.com/rainmanjam/poka-yoke.git /tmp/poka-yoke, then run python3 /tmp/poka-yoke/plugins/poka-yoke/scripts/cli.py detect --paths /absolute/path/to/your/repo. The path has to be absolute, otherwise it resolves against the clone and not your project.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.