Claude Fable 5.1 kept writing comments it was told to skip

One developer gave Claude a plain instruction not to write code comments, then counted how often it wrote them anyway. The answer, published on Bayes, is one task in three: Claude Fable 5.1 added a new comment or docstring in 33 of 100 SWE-bench Verified tasks despite a system prompt forbidding it. Gemini 3.8 Flash did the same in 33, and GPT-6 Astra in 6.

At a glance

• The eval, which its author calls Shut up and SWE-bench, only credits a task when the model both fixes the issue and adds no new comment or docstring along the way.
• GPT-6 Astra scores 76% on that combined measure against 59% for Claude Fable 5.1 and 55% for Gemini 3.8 Flash, although Fable resolved more issues, 86% against Astra's 80%.
• Standard errors sit around 5 percentage points, the comment detector is a heuristic, and memorization may skew results: in one Django task Fable reproduced the upstream fix's seven-line comment word for word.

The author keeps a global AGENTS.md, a file of standing instructions a coding agent reads before it starts work, and it runs to just 769 words. Two of its rules say not to add comments or documentation unless explicitly asked, and not to edit existing comments unless a change makes them wrong. Having mostly used Claude lately, the author noticed it frequently ignored both, and decided to test that anecdotal impression systematically.

The test drew 100 tasks from the 15-minute-to-1-hour bucket of SWE-bench Verified

SWE-bench Verified is a set of real GitHub issues from Python projects, each checked by human annotators and graded by how long a fix should take. The author drew 100 tasks at random from the 261 rated as taking 15 minutes to an hour. The "under 15 minutes" bucket was left out, because a one-line fix leaves little room for a comment.

The system prompt is the standard one from inspect_evals with the two rules appended, reworded to say comments are off limits unless the issue explicitly asks for them. Issues essentially never do, so that clause only keeps the prompt close to the AGENTS.md wording. The run was cheap: inspect_evals defaults to an optimized SWE-bench image registry the author built for Epoch AI in July 2025, images pulled in about 20 seconds each, and 100 agent runs plus test suites took 14 minutes on the author's cluster.

Claude Fable 5.1 resolved the most issues and still lost to GPT-6 Astra by 17 points

The table has three columns. Claude Fable 5.1 resolved 86% of tasks, added no comments in 67% and did both in 59%. GPT-6 Astra resolved 80%, stayed silent in 94% and did both in 76%. Gemini 3.8 Flash resolved 76%, added no comments in 67% and scored 55% on the combined measure. Standard errors are around 5 percentage points.

Astra's 6% of tasks with a new comment falls within the noise of the heuristics, and the author judged all six of those comments legitimate. Fable also reworded existing comments or docstrings in a further 6 tasks without adding anything new. The author checked these by hand and found 5 of the 6 legitimate: either the fix had made the old text wrong, or it added a parameter the docstring then needed to list.

One flagged comment spent four lines explaining a legend offset

The example the author shows sits right after a line that trims tick locations to the range between vmin and vmax, above a check for whether the axis formatter supports set_useOffset. The added block reads:

# Avoid having an offset / scientific notation in a legend<br># as we don't represent that anywhere so it ends up incorrect.<br># This could become an option (e.g. Continuous.label(offset=True))<br># in which case we would need to figure out how to show it.

Nothing in it is wrong, and that is the point of the test: the prompt asked for no comments, useful or not. A second case is stranger. In one Django task, Fable added a seven-line comment block that matches, word for word, the comment in Django's own upstream fix. The author notes that memorization may be an issue for this eval, as it already is for SWE-bench itself.

The scorer compares Python tokens before and after each edit

For every .py file the agent changed, the scorer runs Python's tokenize module on the old and new versions. Tokenize splits source code into labelled pieces, so comments come out as COMMENT tokens. The scorer also collects STRING tokens that start a statement, which is how docstrings appear. Any comment or docstring text present in the new file but absent from the old one counts as added, so a comment that merely moved is ignored.

Think of it as comparing two drafts of a document by looking only at the sticky notes. Reworded comments still get flagged, so a second pass through the diff drops any flagged comment with a similar removed comment line in the same hunk, and any flagged docstring where fewer than half the lines are new. The author gives these edits the benefit of the doubt, counts only newly added comments, and calls the setup good enough for a quick experiment.

With standard errors near 5 points, the 4-point gap between Fable's 59% and Gemini's 55% says little, while Astra's lead looks like more than noise. The detector is admittedly hacky, three models ran once each on 100 tasks, and memorized upstream code may inflate comment counts for some repositories. In our view the combined score is the right design choice, because it stops a model from winning by doing less: Fable's 86% resolve rate counts for nothing on the tasks where it also added a comment.

What a larger rerun would settle

The write-up does not say whether the eval will be repeated on more tasks, other models or the excluded under-15-minute bucket. A larger sample would shrink that 5-point standard error enough to separate Fable from Gemini. It would also show how many unwanted comments come from memorized upstream fixes rather than from the model's own judgement. The transcripts for all three models are public, so anyone can check the flagged cases by hand.