Skip to content

OpenAI

OpenAI wants a shared limit on AI progress

3 min read

Written by an AI model from the linked sources and checked against them by a second model. How we work

All about: OpenAI models and products →
Image: openai.com

OpenAI has published the internal score it has used for years to track how much smarter its models get during training, and it is asking other AI labs to do the same. In a post on OpenAI’s blog, Pachocki, the post’s author, argues that this one number, called perplexity, should become a shared yardstick for the industry. He also wants labs to agree on limits measured by it and not to cross those limits until they can properly understand and monitor what their models do.

He admits the disclosure has a cost: it gives competitors a clearer picture of where OpenAI stands. In his view, the need for safety measures the whole industry shares outweighs that concern.

What the score measures

Perplexity measures how well a model guesses the next piece of text it has never seen. Lower means better guesses. OpenAI tracks this during pretraining, the first stage of building a model, when it learns from a huge pile of text. Pachocki’s argument is that most of what a model can do is learned at this stage. Later techniques such as prompting and fine-tuning mostly draw out abilities that are already there.

He ties past leaps in ability to points on OpenAI’s own scale. GPT-3.5, which the original ChatGPT was built on, scored about 0.99 and could hold a fluent conversation. GPT-4, at about 0.91, could reason usefully step by step, and that ability led to today’s reasoning models. Below 0.8, strong skills in cybersecurity and mathematics appeared. Pachocki stresses that nobody knew in advance these skills would show up at those levels. According to AI Parabellum, the scores come from a test set drawn from OpenAI’s internal codebase, so older models look weaker on it than they would on ordinary text.

Why he wants limits now

Mathematics is the example that worries him. OpenAI has given its models open math problems for more than a year as a way to test them. Once the company moved to a model scoring about 0.76, solutions started arriving one after another in a single Slack thread, which Pachocki calls “deeply surreal” for colleagues who used to work in mathematics. He thinks the same kind of sudden jump could happen once AI starts doing research on pretraining itself, perhaps as early as 2027. He expects AI to help with safety research too, but probably not quickly enough to keep up.

Based on recent progress, he expects more transformative abilities somewhere around 0.68 to 0.7. He doesn’t know what they will be, and labs may not be testing for them yet. He also expects today’s safety audits to get less reliable and models to get better at noticing when they are being tested or trained.

His proposal has three parts. Labs should be open about their perplexity scores. They should agree on shared rules for testing and red-teaming very capable models before those models are first used. And they should commit to common limits, based on perplexity and other measurements, that stay in place until a concrete safety milestone is reached. One example he gives is a successor to the chain-of-thought monitors labs use today to read a model’s reasoning.

Where the yardstick falls short

Pachocki lists the weaknesses himself. Labs train on different data and test on different datasets, so their scores can’t be compared directly yet. The score doesn’t correspond to any real task. It also misses gains made after pretraining: step-by-step reasoning makes models much more capable without changing their perplexity at all. And nobody knows how low the score can go. Near that floor, he writes, progress on the number will look slower even as small gains bring bigger leaps in intelligence.

He hopes the post is a first step toward a common framework for setting limits one after another.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.