GPT-6.1 Sol jumps from 5.4% to 86.6% on COBOL migration
OpenAI said on October 19 that GPT-6.1 Sol ports COBOL programs to C++, Java, Kotlin, Python or Rust with 86.6% accuracy at low reasoning effort, against 5.4% for GPT-5.6 Sol at the same setting and at 39% lower cost, in OpenAI’s post on low-resource languages. The chart data behind the post also puts GPT-6.1 Sol within 0.35 points of the headline model, GPT-6 Astra, on the MultiLangBench database-building test, at a mean $7.53 per sample against Astra’s $32.18. Both are OpenAI’s own evaluations, and only MultiLangBench is public.
COBOL migration by effort level
The migration test counts a program as ported only when every hidden test matches its observable output byte for byte. That bar is aimed at bugs that run without errors: COBOL’s fixed-point arithmetic drops extra decimal digits unless rounding is requested, so a rewrite that rounds them produces different financial results. The stakes OpenAI cites come from an IBM page the post links: COBOL behind more than 40% of online banking systems, 80% of in-person card transactions and 95% of ATM transactions.
Higher effort lifts GPT-6.1 Sol to 97.9%, which OpenAI says nearly saturates the evaluation. The chart shows 97.9% at both xhigh ($0.70 mean cost) and max ($0.98), while medium scores 96.9% for $0.51, above high at 95.9%. That $0.51 is also what GPT-5.6 Sol cost at low effort, for 5.4%.
GPT-6 Astra reaches 96.8% at xhigh and max for $3.26 and $4.51, GPT-6 Sol 95.8% at max for $1.92, and GPT-5.6 Sol tops out at 65.2% at max; the post’s text gives none of these. OpenAI adds that production software still needs expert review and testing against the system’s business rules and dependencies. No dataset or code for the migration test has been released.
Twenty languages, one database engine
MultiLangBench has a model build a SQLite-compatible engine from scratch in a given language over an extended session. The candidate takes a database and a SQL query and writes a modified database, which the reference SQLite checks, so the same tests, adapted from SQLite’s own suite, apply in every language.
At max effort GPT-6 Astra averages 87.4%, up from 26.8% for GPT-5.6 Sol, and clears 83% in 19 of 20 languages; COBOL stays lowest at 72.1%, up from 10.9%. GPT-6 Luna at max effort reaches 40.3% at an estimated $0.77 per sample, against $3.48 for GPT-5.6 Sol, about 78% less.
In the chart data we extracted, GPT-6.1 Sol at max effort averages 87.03% and beats Astra in 7 of 20 languages: C, COBOL (72.6% vs 72.1%), Fortran (84.1% vs 83.5%), Go, OCaml, Python and Scala. Astra’s COBOL sample alone costs $54.72, for the lower COBOL score. After COBOL, Astra’s weakest languages are Fortran, Lean (84.0%) and Scala (84.8%); GPT-6.1 Sol’s are Lean (81.9%), Fortran and Zig (84.9%).
Our analysis
What we checked
On October 20 we cloned the MultiLangBench repo at commit 995755b, counted and hash-checked its tests, and ran a stub candidate that forwards each query to real SQLite 3.40.1 under the README’s grading procedure. We also pulled the per-language and migration figures out of the chart data in OpenAI’s post.
OpenAI’s headline figures match its own chart data, and the suite holds together: real SQLite passes the official subset in full. The post says the setup and test suite are being released along with per-language results, but the repo ships test data and a golden generator only, with no runner for candidates, and the per-language results live in the post’s charts.
| Check | Result |
|---|---|
| Repo contents | Three scripts, README, licence, requirements; none runs a candidate or checks its language |
| Grading rules | 60-second timeout, crash = fail, wrong language = 0: README text only, documented, not enforced in code; README suggests another language model as judge |
| Tests | 23,939, all hashes valid; 272 in the official subset |
| Real SQLite, 272 subset | 272/272 in 13.9 s on one CPU |
| Real SQLite, 1,516 tests | 1,515 pass; degrees(acos(0.5)) gave 60.00000000000001 vs golden 59.99999999999999 on our aarch64 machine, one run |
| Chart data | 440 records; Astra 87.38%, GPT-5.6 Sol 26.77%, Luna 40.30%, Luna 77.9% cheaper |
Scoring a model yourself
Any lab or team reporting a MultiLangBench score writes its own runner, so the timeout, the crash rule and the language check depend on that runner and on whichever model judges the language. The 272-test subset grades in seconds, which leaves the model session as the real cost. On the full suite, at least one golden depends on the machine’s floating-point math, so a correct engine can lose a test there.
Up next
FLUX 3 Max outpaces Seedance 2.5 over 25x in fal’s test
FLUX 3 Max rendered a 10-second video in about 11 seconds in fal’s benchmark, while Seedance 2.5 took 277.8 seconds, and fal says real times vary with settings. How many takes does one usable AI video clip usually cost you?
Related stories
- OpenAI’s half-price GPT-6 Sol and Luna aren’t a promo
- A daily benchmark checks if Codex quietly got worse
- Codex desktop app appears to be dropping Fast mode on API keys
- Microsoft’s MAI Code 2 claims 96% of Opus 5.5 at 5% of price
- Claude Haiku 5.5 spotted a data leak GPT-6.1 Sol missed
- Codex helped fix a 2007 Radeon video bug for Linux 7.5
Comments
No comments yet. Be the first.
Join the conversation
Sign in to leave a comment. It appears after moderation.

