A daily benchmark checks if Codex quietly got worse

Every day, Marginlab runs gpt-5.6-sol inside the stock Codex CLI against a curated subset of SWE-Bench-Pro and checks whether the pass rate has dropped by a statistically significant amount. In the update dated September 8, the status was Nominal: 88% across 50 eval test cases, against an 83% baseline.

The Marginlab tracker says it is powered by MARGIN EVALS, and it has one stated goal: to detect statistically significant degradations in Codex with gpt-5.6-sol on software engineering tasks. Its setup line is “What you see is what you get”. The model runs in Codex CLI as shipped, with no custom agent harness around it.

That harness rule matters. In August, OpenAI’s developer guide showed the same GPT-5.6 Sol going from 13.3% to 38.3% on ARC-AGI-3 when only the scaffolding changed. A tracker with its own wrapper would be partly measuring that wrapper. Running the CLI as users get it ties any movement in the number to the model or to Codex itself.

Detection works by comparing each result with history. The 83% baseline is the tracker’s historical average pass rate. Each new result is checked against a shaded band around that line, and a change inside the band is not statistically significant (p ≥ 0.05). Each point can also show a 95% confidence interval, and the tracker notes that those intervals widen when fewer samples go into a point.

The two charts use bands of different widths. The daily chart rests on 50 test cases a day, and its band is ±11.2% around the baseline. The weekly chart rolls seven days into one figure, 350 test cases, and its band narrows to ±3.6%. The 30-day figure covers 1,500 test cases. So a single day can flag only a large drop, and slow erosion would surface in the seven-day line first.

Right now nothing is close to either edge. The past seven days and the past 30 days both came in at 85%, two points above the baseline and inside the tighter weekly band.

Running a fixed task set every day helps because users’ reports of degradation are hard to separate from noise. Anthropic said as much in its postmortem on three infrastructure bugs that intermittently degraded Claude’s responses: initial user reports were difficult to distinguish from normal variation in feedback. One of those bugs, a context-window routing error, began on August 5 affecting 0.8% of Sonnet 4 requests and hit 16% at the worst hour on August 31. The same document states: “We never reduce model quality due to demand, time of day, or server load.” A fault that starts below one percent of traffic would likely sit well inside even a ±3.6% band, so a tracker like this one is probably better at catching broad drops than partial ones.

The complaints kept coming after the September 8 reading. Later, a GitHub issue from a Windows user on ChatGPT Pro reported that from about mid-September, gpt-6-astra and gpt-6.1-sol at max reasoning effort performed substantially worse on real coding and agent tasks. The reporter said the earlier performance came back briefly, for about two days, and also described “Selected model is at capacity” errors.

On October 10, a thread on the OpenAI Developer Community forum called Codex “practically unusable” and suspected that requests were being routed to a GPT-4o-mini-like model whatever model the user selected. Among its related topics is one dated September 9, the day after the Nominal reading: “We notice the abnormal regression in behavior from the models”.