Skip to content

Anthropic

Rubrik found more bugs than its engineers could fix

3 min read

Written by an AI model from the linked sources and checked against them by a second model. How we work

Image: thenewstack.io

In its first month with Claude Mythos Preview, Rubrik got more vulnerability findings than its engineers could sort through. Instead of hiring more reviewers, the company built software around the model to filter what reaches people. CTO and co-founder Arvind Nithrakashyap walked through the setup in a The New Stack article from August 13.

The findings went beyond what Rubrik’s usual tools and methods catch. Mythos turned up complex vulnerability chains right away, he said. Those included relationships between components across large codebases that neither a conventional scan nor one engineer’s review would have caught. Rubrik got access in June, when Anthropic widened Project Glasswing to roughly 150 organizations across 15 countries. Only vetted partners invited into the program can use Mythos Preview.

“When we first saw the readout of potential issues, our immediate instinct was to treat it as a capacity problem,” Nithrakashyap said. Rubrik considered adding human reviewers to handle the volume, then dropped the idea: “We quickly abandoned the plan as we realized there was no way human-driven remediation could keep pace with AI-speed discovery.”

So Rubrik put together a team from engineering and infosec and had it automate wherever it could. “The focus was to build an effective harness,” he said, one “that manages tool calls and checkpoints, adds business context, security context, and trust boundaries.” The harness is a software layer wrapped around Mythos. Its job is to cut the number of findings that end up in front of engineers for review and remediation.

The pipeline runs in stages. Mythos first scans the whole repository. Those initial findings then shape a series of more targeted passes, which weed out the noise, so only high-quality findings are routed to the appropriate teams and prioritized. Nithrakashyap said the workflow became manageable for engineers only once the targeted passes were in place. A month in, the main answer to a model that outruns human reviewers is a filter placed in front of them.

How far to automate turned out to be its own problem. “Surprisingly, we ran into this question about what not to automate quite often,” he said, and he described a tension between “trustworthy automation” and “maximum automation”. Rubrik chose the first. “To maintain trust, we chose to limit automated remediation to a deliberate, tightly-scoped subset of vulnerability classes where machine-driven fixes are highly reliable and well defined,” he said. Only findings in those predefined classes take the automated path. Everything else Mythos surfaces goes to an engineering team, and a person owns the final fix.

The New Stack reads Rubrik’s month as a sign that most existing engineering workflows likely aren’t ready to keep up with how fast the model finds things. On that reading, adapting means building structural context into the harness and setting up systems that sort and filter the model’s output into findings a team can act on.

“Mythos has shown us that AI actually increases the demand for engineering rigor in the systems that surround it,” Nithrakashyap said.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.