Luna at max effort tops MotherDuck’s SQL benchmark

After OpenAI cut the API price of GPT-5.6 Luna by 80%, Luna running at max effort became the top model on MotherDuck’s agentic SQL benchmark. It holds 99.8% accuracy at one fifth the price of the previous leader, Gemini-3-Flash on low effort. MotherDuck published the results on August 14 in a post on its blog. The team writes that the change showed up in its evals immediately, and that at first “we even thought we had made a mistake.”
The headline figure comes from a different test. On a semantic modeling benchmark, Luna’s cost fell to a tenth of what the same model cost the week before. The price cut alone accounts for a factor of five. MotherDuck attributes the other factor of two to improvements in model caching, and says it will publish the details of that benchmark in a separate post.
The figures and their conditions
Every number in the post is tied to a specific benchmark and effort setting:
- Agentic SQL benchmark: Luna at max effort holds 99.8% accuracy, at a price five times lower than Gemini-3-Flash on low.
- Semantic modeling benchmark: ten times cheaper than Luna a week earlier, with caching credited for half of the gain.
- DeepSWE: Luna scores 3% below Fable at less than a tenth of Fable’s price.
- Full answer cost: MotherDuck says a small model paired with a fast analytical database can return low-latency answers for less than half a penny each, with both AI and database costs included.
MotherDuck compares the size of the shift to the arrival of Opus 4.5, which it says made models practical for professional-sized coding workloads. This time the change happened “at a different spot on the Pareto frontier”: Luna writes SQL well, and “unlike Opus, it is also incredibly fast and cheap,” according to the post.
The team’s previous champion has a public baseline. In its open DABStep harness, MotherDuck reported 419 of 419 questions answered correctly on the cleaned full test set with gemini-3-flash at low reasoning, at about $7.91 per run, or roughly $0.019 per question. The same README notes that low effort matched medium-effort accuracy at half the cost. With reasoning switched off, a run cost $7.30, but adherence varied by about 1%.
How the harness is built
The setup behind these figures is documented in that repository. The agent loop runs on the openai-agents SDK through OpenRouter, the data sits in MotherDuck as DuckDB SQL, and answers are graded with the strict DABStep scorer. The design draws on Anthropic’s self-service data analytics work. At Anthropic, the same approach of Markdown skill files had pushed agents to about 95% accuracy on business queries in June, and accuracy fell to 21% when the skills were removed.
The harness has two parts. A compact skill file goes into the system prompt on every call. Everything else lives in a semantic layer that the agent queries through semantic_lookup, opening it one level at a time: first the domains, then one-line summaries of the items, then the full text. The complete knowledge base runs to about 53K characters, but on a hard question the agent loads only about 16K. The always-on prompt is about 3.4 times smaller than pasting in the full manual.
The repository also records one lesson about cheaper models: rules that carry the most weight belong in the always-on skill and not behind a lookup, “otherwise a low/no-reasoning model skips them.” The blog post makes the same argument from the other side. When weaker models could not write accurate SQL, context only needed to be good enough for the strongest models. Now, the post says, more detailed documentation of the domain pays off because a smaller model can follow it, and that work “can slash costs and speed up answers.”
What MotherDuck suggests changing
The first change is a model swap. Anyone running a large model on low effort should try Luna on max, and tasks already marked as not needing maximum intelligence are the obvious candidates. The second is to ask more questions. MotherDuck calls this the Jevons Paradox, where cheaper technology quickly finds more uses, and gives examples: test five hypotheses at once before the agent reports back, go further into detail, or fetch answers to likely customer questions before customers pick from a list.
The bigger change is in the database. In agent workflows, the LLM call has long been the slowest part of each step. When one agent turn took tens of seconds, a faster database made little difference. Now that the model is fast, MotherDuck argues, a transactional database becomes the bottleneck for the user experience. Analytical engines can be 1000x faster than transactional ones on analytically shaped workloads, and the post says many agent questions are analytically shaped. Startup time counts too: if an analytical store takes 30 seconds to spin up, a low-latency database could have answered 10 agent questions in that time. Agent traffic comes in bursts, so the post recommends a low-latency serverless setup, especially for agent features that customers use directly. MotherDuck sells exactly this kind of serverless analytical database.
Why evals now run more often
MotherDuck applies the same 5x to evaluation. A data team’s tests have usually been data quality checks on simple invariants: no duplicate customer IDs, no NULL order prices, every order joining to a real product ID. Agentic analysis needs a different kind of test: natural-language questions paired with the correct answers from the data, plus a check on whether an agent with business context and a database connection gets those answers right. Running such evals is now five times cheaper, and the post suggests spending the savings on running them far more often.
The post lists several reasons to do so. Frequent runs let a team compare each new model and tune settings within a model, where it might previously have taken one lab’s defaults. They can also catch a model getting less capable without any change to its weights. Performance depends on the serving infrastructure as well, and the post says that “as capacity gets tight, model intelligence is reduced.” A well-calibrated eval setup could catch that and switch to another model, whether in the days before a new release or at peak traffic hours. OpenAI’s own data team found that one of its eval framework’s biggest benefits was catching regressions in its context layer. In MotherDuck’s view, running evals once a week does not give enough signal. The business changes as well: after a new discount program launches, an agent may no longer work out revenue correctly.
MotherDuck runs its evals in a custom harness on OpenRouter, so it can test any frontier lab’s newest model within hours of release. The stated goal is to build intellectual property in the harness and the context layer, treat the model as a commodity, and keep switching costs low enough to move to the next top model once it pays off. Later, on August 28, MotherDuck published a comparison of cost and runtime against accuracy on DABStep for frontier cloud models and local ones, Qwen3.8 27B among them.
The post closes on the price race between the labs: “For data questions, GPT 5.6 Luna legitimately looks like we’ve arrived.”
Up next
27B open weights ship under Apache 2.0
“262K native context, extendable to 1M tokens via YaRN”: the Qwen3.8-27B weights are up on Hugging Face under Apache 2.0, a dense multimodal model small enough to run locally.
Related stories
- Claude Haiku 5.5 spotted a data leak GPT-6.1 Sol missed
- OpenAI drops lottery for free ChatGPT access for scientists
- Fable 5.2 matched Opus 5.5 on nine tasks at almost 4x cost
- Cheaper per token, Fable 5.2 cost $13.39 to Astra’s $3.99
- Opus 5.5 beats GPT-6 Astra on data-analysis repeatability
- GPT-6 Astra hits 66.7% on wet-lab tasks, GPT-5.6 Sol 43.8%
Comments
No comments yet. Be the first.
Join the conversation
Sign in to leave a comment. It appears after moderation.

