Skip to content

Claude Code

Claude Code and two free models split on Next.js picks

6 min read

Our AI analyst checked this story itself: it ran, measured and compared against the originals. What we checked

Written by an AI model from the linked sources and checked against them by a second model. How we work

All about: Claude Code versions and changes →
Image: whatagentspick.com

Claude Code and two free models in OpenCode, DeepSeek V4.1 Flash and GLM 5.3 Flash, landed on the same top tool in 2 of 3 categories for a Python project and in none of 3 for Next.js, per a Whatagentspick scoreboard updated on October 27. On Next.js, the ORM, host and email service an agent installs shift with the model, though each free model ran only six times per cell, and in our recount none of the four splits between Claude and a free model is statistically distinguishable at these sample sizes.

Python agrees, Next.js splits

On the FastAPI project, the three agreed on SQLAlchemy for the database layer (Claude 18 of 18 runs, DeepSeek 6 of 6, GLM 4 of 5) and on raw SMTP for email (17 of 18, 5 of 5 and 5 of 6). Deployment was the one Python split. Claude tied Heroku and plain Docker at 7 of 18, while DeepSeek picked Heroku in 3 of 6 runs and GLM in 4 of 6.

Next.js with TypeScript split in every category. Claude chose Drizzle ORM in 12 of 18 runs and GLM in 5 of 6, but DeepSeek tied Drizzle and Prisma at 2 of 6. For deployment, Claude went to plain Docker in 14 of 18. DeepSeek tied Fly.io and plain Docker at 2 of 5, and GLM tied plain Docker with a “built its own” result at 3 of 6. For email, Claude picked Resend in 11 of 18, DeepSeek tied raw SMTP and Resend at 3 of 6, and GLM chose raw SMTP in 4 of 6.

The board’s two biggest splits run along the stack, for every model. Heroku appeared in 14 of 30 Python deployment runs and in none of 29 on Next.js. Resend took 16 of 30 Next.js email runs and none of 29 on Python, where raw SMTP took 27. In our recount, every raw SMTP detection on Python (32 of 32) came from an import of smtplib, which ships with Python and needs no install. On Next.js, the same pick meant adding the nodemailer package. By our reading, part of the email split comes from that difference as well as from the models’ preferences.

Claude’s column is itself a sum of three models. On Python deployment, Opus 5.5 picked Heroku in 4 of 6 runs and Fly.io in the other two. Sonnet 5.5 went to plain Docker in 3, Fly.io in 2 and Heroku in 1, and Haiku 5.5 chose plain Docker in 4 and Heroku in 2.

The wording picked the host

Deployment and email prompts came in three wordings, and the board pools them. The second deployment wording read: “I want this running on the public internet. Set up the deployment configuration and tell me what to run to ship it.” It produced Fly.io in 6 of 10 scored runs on Python and 6 of 10 on Next.js, and no Heroku at all. The other two wordings produced Heroku in 14 of 20 scored Python runs and no Fly.io. On the second wording for Python, Opus, Sonnet and DeepSeek each went to Fly.io in 2 of 2 runs, while Haiku and GLM went to plain Docker in 2 of 2.

Email moved with the wording too. Resend took 7 of 10 Next.js runs on the first wording, 6 of 10 on the second and 3 of 10 on the third.

Amplifying ran 2,430 open-ended prompts through Claude Code with no tool named in any of them. Testing Sonnet 4.5, Opus 4.5 and Opus 4.6, it reported that the project mattered more than the phrasing, with picks holding across five phrasings at 76% average stability. In these runs, deployment is where the phrasing moved the pick.

How many runs back it?

Claude Code ran 18 times per category and stack, six each for Opus 5.5, Sonnet 5.5 and Haiku 5.5. That made 108 scored runs at a cost of $11.21: $6.64 for Opus, $3.25 for Sonnet and $1.32 for Haiku. The free columns rest on six runs per cell on OpenCode’s free tier, at $0. DeepSeek had 34 of 36 runs scored and GLM 35 of 36, and the 3 of 72 runs that made no attempt are left out. Whatagentspick itself says to read these small samples as a lead.

Claude’s runs are dated October 17 and 18. The free ones ran on October 27 through OpenCode 1.19.4, nine to ten days later. The free-model runs were capped at 12 steps. The project’s README says Claude ran under the same stop rules, which the methodology page describes as a turn, token or time cap. The cap stopped the free models more often: 13 of 36 DeepSeek runs and 13 of 36 GLM runs ended at 12 steps (36% each), against 22 of 108 Claude runs that stopped at their cap (20%). In every run, the detector looks only at changed files and keeps one pick per run, the one with the strongest evidence.

Our analysis

What we checked

On October 28, we recounted the runs from the project’s raw data on GitHub, looked at the detector evidence behind the deployment picks and ran a Fisher exact test, a test built for small counts, on the four splits.

How we check stories

Every pooled number on the board matches the raw runs, but two of its labels measure less than they say: “Heroku” marks a Procfile written next to a Dockerfile, and the “built its own” deployment runs wrote no deployment files. The test cannot separate Claude from either free model on any of the four splits, which means these samples are too small to show a difference; it does not show that there is none.

  • Two-sided p-values for Claude against DeepSeek or GLM run from 0.14 (plain Docker on Next.js, against DeepSeek) to 0.67 (Resend on Next.js and Heroku on Python, both against DeepSeek).
  • Under Resend, the board shows bars of 4/6 for Opus, 4/6 for Sonnet and 5/6 for Haiku. Sonnet’s bar is its own top pick, raw SMTP through nodemailer, and it chose Resend in 2 of 6 runs.
  • All 14 Heroku picks (7 Claude, 3 DeepSeek, 4 GLM) come from runs that wrote both a Procfile and a Dockerfile, and the detector ranks the Procfile higher. Only 8 of those final messages mention Heroku, and 10 also name Render or Railway. Railway, Render and Dokku read a Procfile as well, so by our reading the label stands for a Procfile-style process declaration.
  • The project counts a run as “built its own” when no third-party package for the category was added. GLM’s three such Next.js deployment runs and DeepSeek’s two deployment runs with that label all stopped at the 12-step cap with no deployment evidence, and all three GLM runs named Vercel in their final message.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.