Flaky tests after a 10x speedup, and what Claude got wrong

A rewrite of an integration suite cut a 1360-test run from around 40 minutes to around 4 by executing many tests in parallel, and left a handful of flaky failures behind. Claude removed almost all of that flakiness, though not without wrong turns, according to an account published on Henrikwarne.

At a glance

• The parallel suite reuses instruments that are defined once and shared across many tests, which speeds execution but exposes the tests to interference and timing-dependent failures that disappear when a test runs alone.
• A colleague who had ported many of the test cases reviewed the first batch of commits and questioned several of them: the rationales sounded plausible, but the changes had no effect.
• Claude's contribution ran to analysing stack traces, cross-checking multiple log files and comparing timestamps for overlapping actions, work the author says would have been too time-consuming to justify before LLMs.

The episode is a concrete version of a problem that appears wherever an assistant diagnoses faster than its user can verify. Flakiness is inferred from indirect evidence, so a fix that sounds plausible is hard to falsify, and a suite that fails less often can look repaired while the real race condition remains. The lasting value here sits less in the patches themselves than in the explanations that made review possible.

A rewrite runs 1360 integration tests in about 4 minutes instead of 40

The system under test is covered by integration tests that run against a complete local instance, driving traffic over FIX connections and administration over HTTP. The 1360 tests previously ran sequentially and took around 40 minutes; the rewrite brought a full run down to around 4 minutes.

To run in parallel, the suite relies on instruments that are defined once and then reused across many tests, which leaves the tests open to interference. After the rewrite, a few tests typically failed on each run and passed when executed on their own, the definition of a flaky test.

A colleague's review found fixes that sounded plausible and changed nothing

Each failing test produced a stack trace, but the trace often showed only where a wait for a desired state had timed out, leaving the underlying cause unclear. Claude was good at finding those causes, and the author accepted and committed the proposed fixes without working through the reasoning behind them.

Because the failures appeared at random, it was hard to confirm whether a given problem had actually been resolved. Flaky failures became noticeably fewer but did not disappear. A colleague who had ported many of the test cases reviewed the commits and questioned several of them.

Claude walked back explanations when the fixes were questioned

On the second pass the author asked Claude about every part of a fix he did not follow. Often the explanations held up and taught him more about the framework and the tests; occasionally Claude retreated, with replies such as "So I was overstating it earlier" and "You're right to push on this, and I need to walk back what I said".

Claude's encyclopedic knowledge of Java and associated frameworks helped, the author writes, and he now uses the same method on unfamiliar code: an overview of how a part of the system works, then follow-up questions to drill into specific areas. Because it does so many things well, he adds, it is easy to assume every output is correct.

The remaining one run in ten

Fewer than one run in ten now ends with a flaky test failure, which the author calls much better than before and still unfinished work for Claude and himself. No target for eliminating the remaining failures is given, and the account sets no timeline; the stated takeaway is that judging whether a proposed fix is valid remains the deciding skill.