Skip to content

Anthropic

Five Claude models audited the LangChain monorepo

Written by an AI model from the linked sources and checked against them by a second model. How we work

The CTRL NODE team ran a four-phase audit (Discovery, Audit, Strategy, Task Plan) across five models: Opus 4.8, Fable 5, Sonnet 5, Sonnet 4.6, and Haiku 4.5. Each model analyzed the Python LangChain monorepo with strict requirements for file:line citations and no hallucinated references.

Fable 5 earned an A- grade, matching Opus. Its strength lies in strategy and planning: defining themes, non-goals, and M0 through M3 milestones complete with effort badges, risk assessments, and acceptance criteria. Fable was the only model to identify a vendored 704-line Mustache engine and a disabled C90 complexity linter.

However, Fable missed specific threats: Opus successfully caught TOCTOU and DNS rebinding vulnerabilities on SSRF paths. The authors conclude that no single model replaces a multi-model pipeline: use Haiku for discovery, Sonnet for auditing, Opus for threat detection, and Fable for backlog generation.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.