Skip to content

OpenAI

OpenAI tests whether AI can think like a drug chemist

3 min read

Our AI analyst checked this story itself: it ran, measured and compared against the originals. What we checked

Written by an AI model from the linked sources and checked against them by a second model. How we work

All about: OpenAI models and products →
Video: x.com

OpenAI has released MedChemBench, a test of whether AI models can reason like medicinal chemists, the scientists who design and refine drug candidates. Its 99 problems are drawn from published research. In OpenAI’s announcement on Friday, its newest model, GPT-6.2 Astra, earned 84.5% of the available credit at the xhigh reasoning setting, against 58.2% for GPT-5.6 Sol. OpenAI also states plainly that the results don’t show models can discover new drugs.

This is OpenAI’s second lab-science test this month. On BioBasicsBench, a set of routine wet-lab analyses, GPT-6 Astra earned 66.7% of the available credit.

What the test asks

Drug chemistry is hard because improving one property of a molecule can make another worse. A compound might act strongly against a disease-related protein but be poorly absorbed, or break down too quickly in the body. Chemists work through this by making and testing related compounds, then deciding what to try next. MedChemBench puts a model at a chosen point in a real research program: it gets the target protein, the researchers’ goals, the compounds studied so far and some of the lab results. It then has to rebuild the molecules from drawings, explain the evidence, rank and pick compounds, and predict their measured properties.

In one problem the model sees early results on compounds that block PRMT5, a protein studied in cancer drug discovery. It has to predict how three related compounds will behave: how strongly they block the protein in cells, how easily they cross a cell barrier, and how fast liver enzymes break them down. The later experiments, hidden from the model, showed a tradeoff. Compound 23 was the stronger blocker. Compound 24 crossed the barrier more easily and broke down more slowly.

Even the basics separate the models. Asked to assemble a molecule containing an 18-atom ring, GPT-6 Astra got it right in all ten tries. GPT-5.6 Sol and GPT-5.6 Luna failed every time. One built a 15-atom ring and the other split the molecule into two pieces, so each was analyzing a different compound.

Where the models fall short

GPT-6 Astra earned 95.4% of the credit for rebuilding structures, but only about half for choosing compounds and predicting numerical properties. OpenAI notes that these categories use different tasks and scoring rules. It also lists the test’s limits. Each answer was a single attempt without tools, feedback or new experiments, and the models may already have seen the published sources the problems come from.

Our analysis

What we checked

On November 21 we read the 27-page preprint linked from OpenAI’s post and the data behind the charts on its blog page.

How we check stories

The newest model has only an overall score. The preprint is dated November 9, eight days before GPT-6.2 Astra was announced, and never mentions it. On the blog page GPT-6.2 Astra appears only in the overall chart. The 95.4% and “about half” figures belong to GPT-6 Astra, and nothing OpenAI has published says how its newest model does at selecting compounds or predicting properties.

The weak spot isn’t one model’s. Across the eleven models in the preprint at xhigh, selection scores ranged from 43.58% to 52.78% and prediction scores from 37.69% to 47.02%. Structure scores ranged from 8.74% to 95.37%. The best selection score went to GPT-5.5, an older model. By our rough estimate, about three quarters of the 25-point gap between GPT-6 Astra and GPT-5.6 Sol comes from rebuilding molecules correctly, and almost none from the chemist’s actual choices.

OpenAI has also published two sets of MedChemBench scores that don’t match. Its GPT-6 Astra launch page listed GPT-6 Astra at 49.3% and GPT-5.6 Sol at 47.4%. The preprint gives 83.33% and 58.23%. Neither document explains the difference. That launch page also explains why only GPT models appear: OpenAI says it left out two Claude models because they refuse most of the questions.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.