Skip to content

Claude Code

9.70% of Claude Code artefacts fail to load, census finds

5 min read

Written by an AI model from the linked sources and checked against them by a second model. How we work

All about: Claude Code versions and changes →
Image: toolproof.kynth.studio

Kynth Studios on August 20 published the first edition of a structural census of the Claude Code ecosystem. It read 445,348 skills, subagents, plugins and marketplaces from GitHub and skills.sh on August 14, and found that 43,199 of them, or 9.70%, ship a file an agent runtime cannot register. Nobody has measured this before, so there is no earlier figure to compare it with. The authors say that is the reason they published it.

The census on Kynth’s Toolproof site uses only static checks. Nothing is executed, no model scores anything and no row is curated by hand. The denominator includes everything discovery found, unreadable files too. Passing means only that a file parses into something a runtime could register, which the authors call “a floor, not a verdict”: a listing that passes can still be useless, wrong or dangerous.

Why subagents break more often

Failure rates differ a lot by kind of artefact. Skills fail at 7.89% (27,398 of 347,382) and subagents at 21.9% (15,669 of 71,430). Plugins fail at 0.16% (33 of 20,212) and marketplaces at 1.57% (99 of 6,324). Subagents are about 16% of all listings but about 36% of all failures.

Kynth puts this down to the spec, not to how authors behave. Anthropic’s frontmatter reference marks a subagent’s name and description as Required: Yes, and a skill’s as not required, so a skill with no name falls back to its directory name. “The same careless file is a working skill and a dead subagent,” the census says. Its counts: 4,119 subagents have no name, which leaves nothing to register them under, and 1,799 have no description. The description is the sentence Claude uses to decide when to delegate, so without it there is nothing to route on. Kynth’s own index had this rule the wrong way round until August 2.

One defect covers most of it

38,183 of the failures, 88.4%, come from a YAML block at the top of a file that does not parse. If that header is malformed, nothing downstream can read a name, a description or a tool list from it. Claude Code reads that header at session start to know what a skill is and when to use it, so a broken one means the skill never shows up at all. Kynth describes the publishing path as having no registry gate, no CI convention and no error shown to the author: “the file is committed, the repository is starred, and the artefact silently never loads.”

The other fatal causes are smaller: 128 plugin.json or marketplace.json manifests that are not valid JSON, and 99 marketplaces whose valid manifest lists no plugins. A listing can have more than one cause, so the per-cause counts add up to more than 43,199.

What one listing means

The census counts listings, meaning one artefact in one repository, not distinct artefacts. The 445,348 listings come from 14,233 repositories, 31.3 per repository on average, because the same file gets forked and vendored and is counted once in each repository that carries it. Kynth says the failure rate is the figure to cite, not the population count. The failures are also concentrated: 3,168 repositories, roughly 22% of the total, hold all 43,199 of them. Coverage is limited by GitHub’s search API, which is rate-limited and does not return everything. What discovery truncated is logged, and the population is a large sample of the public ecosystem, not all of it.

All the data is released under CC BY 4.0. broken-listings.csv has 43,199 rows, one per failing listing, each with its repository, path and the cause that failed it. census.json holds the aggregates and the exact failure rule, and listings.csv.gz holds the full population. Edition 2026.1 is archived on Zenodo under a permanent DOI.

A bug in Kynth’s own index

While preparing the census, Kynth found a bug in its own index. A missing-references flag, which marks a skill that names a bundled file not present in its repository, fired on 139,623 listings, 40.2% of all skills. It was wrong roughly seven times in eight, because of three bugs in one regular expression. The pattern listed js before json, so package.json was read as a missing package.js (12,178 listings) and res.json(data) in code samples became a missing res.js (6,946). Matches were not limited to the skill’s own bundle, so the most-reported “missing files” were Next.js (21,014), Node.js (19,383), the skill’s own SKILL.md (27,002) and CLAUDE.md (11,371). And a path written as ./run.sh could never resolve, which flagged 1,498 listings for a file that was actually there.

After the fix, 18,657 skills, 5.37%, really do name a bundled file that is missing. The flag only lowers a score and never decided whether a file loads, so the 9.70% headline is unchanged. The fix is in the open-source scraper these indexes share and reaches the index at the next nightly rebuild. Kynth says the correction is on the census page and not in a changelog because “an index that publishes other people’s broken metadata has to publish its own.”

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.