Skip to content

OpenAI

GPT-6.1 Sol jumps from 5.4% to 86.6% on COBOL migration

4 min read

Our AI analyst checked this story itself: it ran, measured and compared against the originals. What we checked

Written by an AI model from the linked sources and checked against them by a second model. How we work

All about: OpenAI models and products →
Video: x.com

OpenAI said on October 19 that GPT-6.1 Sol ports COBOL programs to C++, Java, Kotlin, Python or Rust with 86.6% accuracy at low reasoning effort, against 5.4% for GPT-5.6 Sol at the same setting and at 39% lower cost, in OpenAI’s post on low-resource languages. The chart data behind the post also puts GPT-6.1 Sol within 0.35 points of the headline model, GPT-6 Astra, on the MultiLangBench database-building test, at a mean $7.53 per sample against Astra’s $32.18. Both are OpenAI’s own evaluations, and only MultiLangBench is public.

COBOL migration by effort level

The migration test counts a program as ported only when every hidden test matches its observable output byte for byte. That bar is aimed at bugs that run without errors: COBOL’s fixed-point arithmetic drops extra decimal digits unless rounding is requested, so a rewrite that rounds them produces different financial results. The stakes OpenAI cites come from an IBM page the post links: COBOL behind more than 40% of online banking systems, 80% of in-person card transactions and 95% of ATM transactions.

Higher effort lifts GPT-6.1 Sol to 97.9%, which OpenAI says nearly saturates the evaluation. The chart shows 97.9% at both xhigh ($0.70 mean cost) and max ($0.98), while medium scores 96.9% for $0.51, above high at 95.9%. That $0.51 is also what GPT-5.6 Sol cost at low effort, for 5.4%.

GPT-6 Astra reaches 96.8% at xhigh and max for $3.26 and $4.51, GPT-6 Sol 95.8% at max for $1.92, and GPT-5.6 Sol tops out at 65.2% at max; the post’s text gives none of these. OpenAI adds that production software still needs expert review and testing against the system’s business rules and dependencies. No dataset or code for the migration test has been released.

Twenty languages, one database engine

MultiLangBench has a model build a SQLite-compatible engine from scratch in a given language over an extended session. The candidate takes a database and a SQL query and writes a modified database, which the reference SQLite checks, so the same tests, adapted from SQLite’s own suite, apply in every language.

At max effort GPT-6 Astra averages 87.4%, up from 26.8% for GPT-5.6 Sol, and clears 83% in 19 of 20 languages; COBOL stays lowest at 72.1%, up from 10.9%. GPT-6 Luna at max effort reaches 40.3% at an estimated $0.77 per sample, against $3.48 for GPT-5.6 Sol, about 78% less.

In the chart data we extracted, GPT-6.1 Sol at max effort averages 87.03% and beats Astra in 7 of 20 languages: C, COBOL (72.6% vs 72.1%), Fortran (84.1% vs 83.5%), Go, OCaml, Python and Scala. Astra’s COBOL sample alone costs $54.72, for the lower COBOL score. After COBOL, Astra’s weakest languages are Fortran, Lean (84.0%) and Scala (84.8%); GPT-6.1 Sol’s are Lean (81.9%), Fortran and Zig (84.9%).

Our analysis

What we checked

On October 20 we cloned the MultiLangBench repo at commit 995755b, counted and hash-checked its tests, and ran a stub candidate that forwards each query to real SQLite 3.40.1 under the README’s grading procedure. We also pulled the per-language and migration figures out of the chart data in OpenAI’s post.

How we check stories

OpenAI’s headline figures match its own chart data, and the suite holds together: real SQLite passes the official subset in full. The post says the setup and test suite are being released along with per-language results, but the repo ships test data and a golden generator only, with no runner for candidates, and the per-language results live in the post’s charts.

CheckResult
Repo contentsThree scripts, README, licence, requirements; none runs a candidate or checks its language
Grading rules60-second timeout, crash = fail, wrong language = 0: README text only, documented, not enforced in code; README suggests another language model as judge
Tests23,939, all hashes valid; 272 in the official subset
Real SQLite, 272 subset272/272 in 13.9 s on one CPU
Real SQLite, 1,516 tests1,515 pass; degrees(acos(0.5)) gave 60.00000000000001 vs golden 59.99999999999999 on our aarch64 machine, one run
Chart data440 records; Astra 87.38%, GPT-5.6 Sol 26.77%, Luna 40.30%, Luna 77.9% cheaper

Scoring a model yourself

Any lab or team reporting a MultiLangBench score writes its own runner, so the timeout, the crash rule and the language check depend on that runner and on whichever model judges the language. The 272-test subset grades in seconds, which leaves the model session as the real cost. On the full suite, at least one golden depends on the machine’s floating-point math, so a correct engine can lose a test there.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.