Opus 5 tops a new conceptual reasoning index at 73.6

Redwood Research and Anthropic on August 12 released the Conceptual Reasoning Index (CRI). It combines three benchmarks for questions that have no answer a model can check its work against. As of August 10, Opus 5 leads with 73.6 (95% confidence interval ±2.1). The authors put the ceiling at about 91, which leaves the best model roughly 17 points short.
The post on Anthropic’s alignment site lists six authors. Emery Cooper, Caspar Oesterheld, Chi Nguyen and Alex Kastner are from Redwood Research, and Joe Benton and Ethan Perez are from Anthropic. Live scores and the methodology are at conceptualreasoning.ai, which the team says it will update as new models and benchmarks come out. The main dataset, LMCA, is available on request through a form.
Why measure it at all
The argument starts from how models are trained. Training depends on plenty of data and reliable feedback, so models tend to do worse on tasks that cannot be verified empirically or mathematically. The authors say much of the work of reducing AI risk is that kind of task. It means reasoning about systems more capable than any human, with no obvious reference class. Some mistakes, such as an AGI takeover or an AI-assisted coup, might only become visible when it is too late. Choices about research agendas and governance play out over years. And some questions, like which values AIs should have, may have no ground truth at all, though the authors still think arguing about them gets somewhere.
They call reasoning under those conditions conceptual reasoning: limited evidence, no answer you can practically verify, and a heavy reliance on argument. A footnote cites METR’s Frontier Risk Report for February to March 2026, in a section titled “Agents had significantly worse judgment and reliability than human experts”, and quotes its observation that “agents were exceptionally strong on problems that are easily “hill-climbable” — those where progress is cheap to verify and many approaches can be tried cheaply”.
The post also explains why timing matters. Once models can do risk-reduction work at the level of human experts, AI-assisted output could dwarf what humans produce alone. So a major factor is how early that point arrives compared with high-risk capabilities. Selectively improving the relevant skills is one way to move it earlier, and improving a skill requires measuring it first.
LMCA, 60% of the index
LMCA (Language Model Conceptual Argumentation) avoids the problem of verifying conclusions by scoring judgments about arguments instead. It contains 560 position texts on decision theory, philosophy and risks from advanced AI, plus 1,461 arguments against those positions. The model rates each argument using a detailed rubric, and its ratings are compared with the experts’ ratings.
Those expert ratings come almost entirely from one person. Emery Cooper rated nearly every argument. The 16 she did not rate were rated by Caspar Oesterheld, and some arguments got independent ratings from at least one more researcher, for 2,140 ratings in total. When two or more humans rated the same argument, they agreed with each other much more than models agreed with humans. The validation set has roughly 50 arguments. Each was rated independently by 4 to 6 people and then discussed for 7 to 8 hours in total.
The same data can also test whether a model can write arguments. Model A writes a new argument against a position text that already has three rated arguments. Model B then gets the rubric and those three rated examples in its prompt and rates the new argument. The authors say this produces fairly accurate ratings. For now, only judging goes into the CRI. Measuring argument-writing is something they hope to add later.
ACCoRD: consistency over correctness
ACCoRD (Assessment of Consistency in Conceptual Reasoning Domains) doesn’t check whether a model’s beliefs are right. It checks whether they fit together logically. Each question asks for a numeric probability or a preference ordering, and the questions go to separate instances of the same model. If one instance gives P(A) and another gives P(A&B), the first number should not be lower than the second. The authors treat inconsistency on a set of questions as a sign that the model’s reasoning on that set can’t be trusted by default.
The full dataset has close to 14,000 consistency constraints across 18 types. Models generated them, and an automated pipeline checked them. The researchers then reviewed and approved 567 of these by hand, and only those 567 count toward the CRI. That is about 4% of the generated pool.
DTBench, close to spent
DTBench capabilities (Decision Theory Benchmark) has 407 handcrafted multiple-choice questions. They cover situations where someone faithfully predicts the model’s own behavior, or where the model interacts with copies or near-copies of itself. Caspar Oesterheld, who has published on decision theory, wrote most of the questions, and almost all of them are original. Emery Cooper validated every one independently. The full DTBench suite has 130 more questions on a model’s decision-theoretic attitudes, which are left out of the CRI.
This benchmark is already near its ceiling: Claude Fable 5 answers 98% of the questions correctly. That leaves the index two components with room to grow, and they carry 80% of the weight between them.
Where the scores sit
The CRI is a weighted average: 60% LMCA, 20% ACCoRD and 20% DTBench capabilities. Every model ran at its maximum token limit and effort level. Zero means random guessing and 100 means the best possible score on every benchmark. On ACCoRD, 100 means perfect consistency. On DTBench, getting every answer right should give 100 or very close to it. LMCA works differently. Human ratings are noisy, so even a model that rates arguments as well as possible should score around 85, based on how often the experts agreed with each other. That cap is why the index ceiling is about 91 and not 100.
The chart shows Anthropic’s best models and the top-scoring model from each other company the team evaluated. It also includes Claude Fable 5, Muse Spark 1.2 and Gemini 3.6 Flash, which are their companies’ strongest models on many outside benchmarks but not on the CRI. Fable 5 refused some questions, and for those the authors used Opus 5’s answers as a fallback. The older data has a gap too: GPT-4 refused to fully answer 18% of ACCoRD items, so its score there is based on incomplete data.
Each lab’s most capable model at release has scored roughly linearly higher since late 2024, and the authors see no sign of the trend flattening. Opus 5 and the other top models on LMCA and ACCoRD are still well below those benchmarks’ estimated ceilings. The team plans to add benchmarks, retire saturated ones and possibly change the weights. Extrapolating from current scores, they loosely estimate LMCA will start saturating about a year from now. The third component is harder to call: “We’re very uncertain about when ACCoRD will saturate.”
Up next
DeepSeek Chat’s Expert Mode now runs on V4-Pro
DeepSeek Chat’s Expert Mode now routes to V4-Pro: 1.6 trillion total parameters, 49 billion active, and the model is available through the API as well. The GA build is labeled V4 Pro 0813 and went live August 12, 2026.
Related stories
- Claude Fable 5.2 hits 69% on PCB layout test, up from 25%
- METR clocks Anthropic benchmark gains at 1.9x old pace
- Sonnet 5 aligned an early Opus 4.8 checkpoint
- Claude Fable 5.1 cracks Urquhart’s Cyphral Distich
- The last of Wiedijk’s 100 theorems falls to Anthropic
- Anthropic paper: automated researchers fix 10/10 benchmarks
Comments
No comments yet. Be the first.
Join the conversation
Sign in to leave a comment. It appears after moderation.

