Skip to content

Community projects

Flaky tests after a 10x speedup, and what Claude got wrong

4 min read

Written by an AI model from the linked sources and checked against them by a second model. How we work

All about: Coding agents →
Image: henrikwarne.com

A developer used Claude to clear most of the flaky tests out of his team’s newly parallelised integration suite, and he had to throw out a batch of commits first. In an August 15 post on Henrikwarne, he describes two passes. On the first he accepted Claude’s fixes without understanding them. On the second he questioned each fix until it held up.

The flakiness came from a speedup. The team runs the complete system locally, and its 1360 integration tests talk to it over FIX connections for traffic (FIX is the messaging protocol trading systems use) and over HTTP for administration. Run one after another, the suite took around 40 minutes. A rewrite brought that down to around 4 minutes by running many tests in parallel. That only works because instruments are defined once and reused across many tests, and shared instruments let tests interfere with each other. After the rewrite, a typical run had a few failing tests that passed when run on their own. That is the author’s working definition of a flaky test: it passes in isolation but can fail on timing when it runs inside the larger suite.

The failures were hard to read. Each one left a stack trace, but often the trace only showed where a wait for some expected state had timed out, so the underlying cause was not obvious.

The author admits he did not know the test framework well, and Claude was very good at finding causes. “So I got carried away,” he writes, and he committed the proposed fixes “without really understanding the rationales.” Because the failures showed up at random, it was not always easy to tell whether a fix had actually solved the problem. The flaky failures became noticeably rarer but still did not disappear completely. Then a colleague who had ported many of the test cases reviewed the changes and questioned many of the commits. Several “sounded quite plausible, but had no effect.” The author realised he did not understand most of the fixes and had simply accepted whatever Claude suggested.

He started over. This time he asked Claude about every part of a fix he did not follow. Often Claude turned out to be right, and he learned how the tests and the framework actually worked. But questioning a solution sometimes got a different kind of answer:

“You’re right to push on this, and I need to walk back what I said”

Other replies he quotes include “Fair challenge — let me be honest about what I can and can’t demonstrate” and “So I was overstating it earlier.” Eventually, he writes, the joint effort would arrive at a solution, and not accepting suggestions blindly kept pointless code out of the tests.

None of this makes him dismissive of the tool. In many cases, he writes, Claude “did a fantastic job” of finding the cause. Analysing stack traces is easy for it, and so is cross-checking several log files and comparing timestamps to spot actions that overlapped. Its broad knowledge of Java and the frameworks around it helped too. He sees the risk in exactly that: because Claude is so good in so many cases, it is easy to assume all of its output is correct.

He has also come to value Claude’s explanations of the existing code in general, beyond any one proposed fix. He now asks for an overview of how some part of the system works and then drills down with follow-up questions, and says he understands existing code much faster that way. He also doubts that, before LLMs, chasing down every cause of flakiness would have been worth the effort, even if the end result is good to have. Work that was not worth doing before, he argues, is now cheap enough to do. His conclusion: if you care about your system, being able to tell whether a proposed fix is valid is very valuable.

Fewer than one run in ten now ends with a flaky test failure. That is much better than before, the author writes, “but still some work for Claude and me to do.”

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.