Skip to content

OpenAI

Luna 6 reports 32x fewer reasoning tokens on ChatGPT Pro

4 min read

Written by an AI model from the linked sources and checked against them by a second model. How we work

All about: OpenAI models and products →
Image: github.com

A community experiment published on October 6 sent identical Codex requests over two routes and got very different reasoning token counts back. On its bookstore task, Luna 6 at Medium thinking averaged 96.30 reported reasoning tokens per response with an API key and 3.025 with a ChatGPT Pro sign-in. That is about 32 times more on the API side, across 40 completed responses per sign-in, according to the study and code on GitHub.

Both routes got every decision on that task right.

The setup is built so that the only things that can differ are the credential and the server it reaches. Every response starts a fresh Codex session. Both runs use the same Codex client, the same requested model and the same thinking level. A local proxy sits between the client and OpenAI and checks that each matched pair sends byte-identical model input. It does not modify the server’s real responses. The authors log the answers and usage counts the server returns, then check them against Codex’s own records. Wrong answers stay in the dataset.

Codex supports both sign-in methods officially, and the choice already affects more than authentication. OpenAI’s documentation says that “when you sign in with an API key, Codex uses standard API pricing instead of included ChatGPT plan credits,” and that data handling follows either the ChatGPT workspace settings or the API organization’s settings. Cloud tasks, GitHub PR reviews and the Slack and Linear integrations work only with ChatGPT sign-in. The CLI, the IDE extension, the SDK and codex exec work with both.

The full dataset is 960 completed responses on two tasks: choosing projects within a budget, and diagnosing duplicate emails from a bookstore. The authors ran Luna 6 and Sol 6.1 at Low, Medium and High, with 40 responses per task, model, level and sign-in. They first collected 480 responses, then ran a fixed second batch of another 480 and analyzed it separately. Four differences passed the multiple-comparison correction in both batches, all on the bookstore task. These are average reported reasoning tokens per response:

  • Luna 6, Low: 68.550 via API key, 0.000 via ChatGPT Pro
  • Luna 6, Medium: 96.300 via API key, 3.025 via ChatGPT Pro
  • Luna 6, High: 148.525 via API key, 85.200 via ChatGPT Pro
  • Sol 6.1, High: 216.675 via API key, 138.700 via ChatGPT Pro

At Low, the subscription route reported no reasoning tokens for Luna 6. At High the gap narrows to about 1.7 times for Luna 6 and about 1.6 times for Sol 6.1, but it stays in place.

The extra tokens did not translate into better bookstore answers. All 480 bookstore decision sets were correct. The project task is the one place where accuracy may have diverged. Across the full dataset, Luna 6 at Low scored 35/40 through the API and 18/40 through the subscription. That pooled difference passed the planned conservative correction (p = 0.0083), but the authors still label it exploratory. On its own, the second batch scored 19/20 versus 12/20, which did not pass its correction (p = 0.371).

The follow-up batch also ran into two capacity failures on the subscription route. The authors kept both in the record, documented how they continued the runs manually and completed the original schedule. The statistics cover completed responses only.

The authors list the limits themselves. The study used two fixed prompts, one API credential and one Pro account. The figures are the usage the server reports, and they don’t reveal the model’s internal reasoning budget. Anyone with Python 3.12 or later can recompute the published results from the included data without calling a model, using python3 -m experiment verify and then python3 -m experiment analyze. The report is written to output/reports/followup/report.md.

Other people have reported similar gaps. A Codex GitHub issue compares GPT-6 Astra through Enterprise sign-in with the public API. The repository’s related-work review also covers a Luna report, bugs in the effort setting and studies that found different results. The authors say none of these explains their numbers. In September, a separate informal test found that some ChatGPT accounts got noticeably weaker Astra output, even though the server still reported gpt-6-astra.

The authors are asking for independent runs on different accounts, plans and collection dates, and they say runs that find no difference are useful too. “We still don’t know why the counts differ,” they write.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.