GPT-6.1 Sol jumps from 5.4% to 86.6% on COBOL migration

On October 19 OpenAI published two coding evaluations for COBOL, Fortran, Visual Basic and similar languages, and its newest models make large gains on both. On COBOL migration, GPT-6.1 Sol scores 86.6% at low reasoning effort, against 5.4% for GPT-5.6 Sol at the same setting.

OpenAI’s post calls these low-resource languages because models have far less code and documentation to learn from than they do for Python or JavaScript. The software written in them still runs essential services. OpenAI says COBOL supports more than 40% of online banking systems, 80% of in-person credit card transactions and 95% of ATM transactions, and that Fortran is used in weather forecasting and nuclear reactor safety analysis. The company says the work grew out of feedback from developers and enterprise customers that coding models struggle with these languages. Those systems often come with incomplete documentation, original developers who have left, and business rules that exist only in the code.

What counts as a migration

The migration evaluation gives a model a COBOL program and asks for a version in C++, Java, Kotlin, Python or Rust that does the same thing. Hidden tests compare the observable output byte for byte, and a migration counts only if every test passes. One mismatched output fails the whole program.

OpenAI’s example of what goes wrong is arithmetic. COBOL works in fixed-point decimals, with a set number of digits after the point, and it discards the extra digits unless the code explicitly asks for rounding. A rewrite that rounds instead will run without errors and produce different financial results.

GPT-6.1 Sol reaches its 86.6% at low effort while costing 61% of what GPT-5.6 Sol costs at that setting, or 39% less. At higher reasoning effort it scores 97.9%, which OpenAI says brings the evaluation close to saturation.

A SQLite in 20 languages

The second evaluation, MultiLangBench, asks a model to build a SQLite-compatible database engine from scratch in a given language over an extended coding session. To run the same tests in every language, OpenAI uses database files as the interface: the candidate program takes an initial database and a SQL query and writes out a modified database, and the reference SQLite checks whether the contents are right.

The GitHub repository released with the post fills in the setup. The model has to implement SQLite 3.40.1 from its documentation, with no internet access and with every trace of SQLite (libraries, executables, embedded implementations) removed from the environment. It must deliver an executable that takes a database path as a command-line argument and reads the query over stdin.

The tests come from SQLite’s TCL suite, and 24k of its 51k tests could be converted. The repo says evals should use only the 272 tests whose names start with 00, 01 and 02; running the full suite is acceptable but should be noted. A test fails on a mismatch, a crash or a 60-second timeout, and a solution not in the requested language gets a pass rate of 0. Scores are reported as a pair of API cost and test pass rate.

At maximum reasoning effort, GPT-6 Astra averages 87.4% across the 20 languages, against 26.8% for GPT-5.6 Sol, and improves in every one:

• COBOL: 10.9% to 72.1%
• Fortran: 21.4% to 83.5%
• OCaml: 25.6% to 85.8%
• Visual Basic: 27.5% to 89.4%

Astra scores above 83% in 19 of the 20 languages. COBOL is the exception at 72.1%, and OpenAI calls it the hardest language in the set.

The two headline results come from different models: Astra for building new code, GPT-6.1 Sol for migration. A chart of MultiLangBench averages published by CellCog puts GPT-6.1 Sol at 87.0 there too, next to Astra’s 87.4, with GPT-6 Sol at 57.7, GPT-5.6 Terra at 37.9 and the older GPT-5.5 and GPT-5.4 at 25.8 and 17.4.

GPT-6 Luna is the cheaper option. It also beats GPT-5.6 Sol in every language and reaches a 40.3% average at maximum effort, at an estimated $0.77 per sample against $3.48 for GPT-5.6 Sol, about 78% less. The per-language gains are smaller: COBOL moves only from 10.9% to 17.1%, Fortran from 21.4% to 31.4%, OCaml from 25.6% to 34.1% and Visual Basic from 27.5% to 41.6%.

The repo also publishes results for each language, so teams can check the language they maintain instead of the 20-language average. OpenAI says it wants models to handle more of the engineering work in more languages: understanding unfamiliar repositories, planning changes, using build tools, running tests and debugging failures. It is asking API users to say which languages and workflows it should work on next.

The company also puts its own limit on the migration numbers. Behavior-preserving migration is useful for teams modernizing older systems, OpenAI writes, though applying it to production software “still requires expert review and testing against the system’s business rules and dependencies.”