Skip to content

OpenAI

A one-line fake proof passes OpenAI’s math Comparator

2 min read

Written by an AI model from the linked sources and checked against them by a second model. How we work

All about: OpenAI models and products →
Image: the-decoder.com

Comparator, the checker attached to OpenAI’s math release, accepts a one-line fake proof of the five-coloring challenge once the submitter redefines a proper coloring as False, the author of an October 9 post on Ohaithe says. The author confirmed the trick, counts six other challenges broken the same way, and says they hadn’t seen the problem discussed anywhere else.

The hole sits in the .json file that comes with a challenge. Names listed under definition_names can be freely overridden by a submission, and the author found definitions that should stay fixed in that list. When that happens, a submission can swap in whatever definition it likes and Comparator still accepts the proof. Lean, the proof language involved, has a computer check every logical step, but it checks the proof against the definitions it is given.

The author’s example is EuclideanFiveColor.lean. The whole file is one definition and one theorem. ProperColoring says a coloring of the complex plane is proper if any two points at distance 1 get different colors. no_proper_five_coloring claims that no proper coloring with five colors exists, and its proof is left as sorry, Lean’s placeholder for a missing proof. The .json lists OAI.EuclideanFiveColor.no_proper_five_coloring under theorem_names and OAI.EuclideanFiveColor.ProperColoring under definition_names.

So the author replaced the definition. The new ProperColoring is just False. Once nothing counts as a proper coloring, the theorem follows in one term, fun ⟨_, h⟩ => h, and Comparator accepted it.

The author concedes this case is easy to check by eye, because the definition sits at the top of a file that imports only Mathlib. Among the six other challenges where a similarly trivial proof gets through, the post names Bernier, LogspaceEquality (the claim that L=RL=BPL), KServer, Naimark and OccupiedOverlapRokhlinSpinAngle. The last one has a legitimate sorry inside a definition, but the author describes it as a proof obligation, not a definitional hole to fill. Comparator still lets the rest of that definition be changed, and two other definitions in its list carry no sorry at all.

The author sees no pattern in which definitions got exposed, and notes that often “only a few definitions out of a large file are given this special, broken, treatment.”

One challenge uses the field as intended, in the author’s reading. ElementaryPositivity has a definitional hole, and the author calls that correct, since its main objective is a data-carrying term instead of a Prop.

OpenAI’s openai/math README says about 42% of top-line results are formalized and points readers to the Comparator instructions for additional checking.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.