Skip to content

Benchmarks

Claude, GPT, Gemini install Sentry and Playwright every time

4 min read

Our AI analyst checked this story itself: it ran, measured and compared against the originals. What we checked

Written by an AI model from the linked sources and checked against them by a second model. How we work

All about: Coding agents →
Image: installrate.com

InstallRate’s leaderboard, run on November 2, gave Claude, GPT and Gemini 600 build tasks that never name a product. In two of its 20 categories, every answer under all three models picked the same tool: Sentry for error monitoring and Playwright for end-to-end tests. Most differences between models below those two sit within sampling noise. Each model got 10 answers per category, made of 5 prompts run twice, and by our recount the column that names one pick per model hides a tie in 9 of the 20 categories.

Stripe took 80% of payments answers under every model, PostHog 93% of product analytics, Neon 67% of Postgres hosting and shadcn/ui 63% of UI component libraries. The site also tracks products the models name and then skip. Datadog was named 9 times and installed 0 times. Radix UI was named 11 times while shadcn/ui got the install. Cloudflare R2 was named, then the code installed the AWS SDK. Vercel took 12 deploy picks to none for Netlify, Railway, Render and Fly.io. A mention counts anywhere in the answer, alternatives and code comments included, and each of those four hosts drew two mentions, all from Claude.

Where Claude goes its own way

Claude put Vercel Flags first in 3 of 10 feature-flag answers. Gemini went to PostHog flags in 5 of 10, and GPT split evenly between LaunchDarkly and PostHog flags. For logging, Claude chose OpenTelemetry in 5 of 10 answers, while GPT chose it once and leaned on Grafana, Loki and Prometheus at 60%. Neither gap passes a Fisher exact test (p = 0.141 for OpenTelemetry, 0.35 for PostHog flags). That test also assumes the answers are independent, and repeats of one prompt are not: the two runs of the same prompt gave the same first pick in 264 of 300 pairs.

What “installed” means here

Per the site’s methodology page, the answers are one-shot API chat completions from anthropic/claude-sonnet-5.5, openai/gpt-5.6-terra and google/gemini-3.1-pro-preview. Each model is told it is a coding agent and runs at temperature 1.0, with up to 2,000 output tokens. The pick is read from the answer’s text. In our count, 494 of the 600 picks came from an install command, 19 from a CLI command and 6 from an import. A small model labelled 24 more, and 56 answers had no pick at all, 26 of them from Claude. The site says API answers “are not the same as a full agent working in your IDE (Claude Code, Cursor, Codex)” and plans agent runs next. Some picks do carry over to an agent harness: Drizzle took 53% of ORM answers here and Resend 53% of email, and in October a different public scoreboard had Claude Code leaning toward Drizzle and Resend on Next.js.

Our analysis

What we checked

On November 5 we recounted each model’s first pick per category from the run’s raw answers. We read the leaderboard against its category and methodology pages and ran two-sided Fisher exact tests on the gaps between models.

How we check stories

The unanimous picks and the named-but-not-installed counts hold up. The per-model column does not. Of 19 gaps between models that we tested, only one falls outside sampling noise: pgvector, picked by GPT in 6 of 10 answers and by Gemini in none (p = 0.011). Treat that as an inference, because the doubled prompts do not fully meet the test’s independence assumption. The leaderboard also says Supabase is mentioned but never installed. That is true of Postgres hosting (4 mentions, 0 installs) but not of the run as a whole: Supabase Auth was installed in 3 auth answers (GPT 2, Gemini 1) and Supabase Realtime in 4 realtime answers, all from GPT.

CategoryModelLeaderboard showsRaw count of 10
Feature flagsGPTLaunchDarkly 30%LaunchDarkly 3, PostHog flags 3
RealtimeClaudeAbly 20%Ably, ws, Socket.IO, Django Channels, no install: 2 each
RealtimeGPTSocket.IO 40%Socket.IO 4, Supabase Realtime 4
Vector databaseGeminiPinecone 20%no vector store 3; Pinecone, Chroma, Qdrant 2 each; pgvector 0
Headless CMSGeminiGit-based CMS 30%Git-based CMS 3, Sanity 3
ORMGeminiDrizzle 40%Drizzle 4, Prisma 4

Gemini never mentioned pgvector in a vector-database answer, so the finding “pgvector beats Pinecone, Qdrant and Chroma” comes from Claude and GPT. Background jobs is a dead heat that the leaderboard page leaves out: Celery ties BullMQ at 12 of 30 overall and at 4 of 10 under each model. The prompt’s stack decides it, the same way for every model. Each of the two Node/TypeScript prompts got BullMQ in all six of its answers, and the Django and FastAPI prompts got Celery in all six.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.