Sai claims a top OSWorld 2.0 score at lower cost

When OSWorld 2.0 told agents to play Chrome Dino, a repetitive real-time game, and clear a score target on a live page, Simular’s Sai agent wrote a program instead of clicking. It measured the ground line, the obstacle speed and its own input latency from raw pixels. Then it wrote a control loop that captures the screen, detects obstacles and jumps, and ran that loop on the VM in a single execute call.
Simular says this habit of turning work into code explains the 73% success rate Sai posted on OSWorld 2.0 in a benchmark update released on August 27, The New Stack reported. The company puts Sai ahead of GPT-5.6 Sol at 62.57% (as reported by OpenAI) and Opus 5 at 70.57% (as reported by Anthropic), and says Sai got there at about 2/3 the cost of either.
What the 73% measures
OSWorld 2.0 comes from the Executable Language Grounding (XLANG) Lab, part of the HKU NLP Group at the University of Hong Kong. It launched on June 26 and was updated on August 8. Its 108 tasks are everyday professional jobs that typically take a skilled human more than 1 hour. OSWorld 1.0 measured its tasks in minutes and 2.0 measures them in hours. The tasks involve finding and reasoning across several data sources (receipts scattered across email and expense reports), reacting when the environment changes (a message arriving partway through), following a tutorial exactly (reimbursement guidelines) and sorting out contradictory data.
Task 28, vaccine booking, is typical. The agent gets an email about required immunizations, a scanned vaccination record on the desktop and a booking site with price, distance and date constraints. Simular’s team calls OSWorld 2.0 the most robust open benchmark in the industry right now and says it reflects real-world tasks better than any other.
Against those tasks, Sai’s lead over GPT-5.6 Sol is roughly 10 percentage points, and its lead over Opus 5 is only about 2. Every one of the three scores comes from its own vendor: OpenAI for Sol, Anthropic for Opus 5, Simular for Sai. Sai’s result isn’t on OSWorld 2.0’s official site yet, though Simular’s OSWorld 1.0 results were listed there. When The New Stack asked why, Simular said:
“It’s not uncommon for companies or organizations to publish benchmark results on their websites first (see Anthropic and OpenAI). We are in the process of submitting to the OSWorld 2.0 leaderboard, and also uploading our trajectories to Hugging Face.”
Simular has been at the top of this benchmark before. Its open-source Agent S was the first to beat the human baseline on OSWorld 1.0, scoring 72.6% in December 2025. That was a benchmark of minute-long tasks. The new 73% comes from tasks that take an hour or more. Later, on August 28, Simular’s site listed a research post titled “Sai tops OSWorld 2.1, beating GPT and Opus with lower costs.”
Where the savings come from
Simular calls its approach neurosymbolic. A neural model does the open-ended exploration, and symbolic code supplies the precision. Once Sai solves a task, the solution is saved as reusable code that replays the same way every time, so the hundredth invoice costs less to process than the first. The planning language is Simulang, which the source describes as a Claude Code skill for desktop automation on macOS. Sai uses it to plan and run longer subtasks, taking more actions per turn. Simular says this gives it about 1.5x fewer model calls on average than pure models.
Sai also controls how much context it sends. Adaptive summarization keeps the input size bounded. Sai keeps the prompt prefix identical for as long as possible before it summarizes, so model providers can serve the repeated prefix from cache. Because the plan lives in code, the important facts about a task stay in runtime memory for the whole run and don’t have to be carried in the prompt.
For orchestration, Sai sends specialized models and interfaces to three jobs: finding UI elements on screen, pure reasoning and verification. It saves the expensive models for steps that need the full task context. Simular says the result is the best outcome-to-cost tradeoff on the benchmark.
On its website, Simular makes bigger claims. It says the approach saves agents “90%+ token usage” and that recurring tasks “get more reliable and affordable over time.” For OSWorld 2.0 itself, the reported figures are about 1.5x fewer model calls and roughly two-thirds of the cost. The cost comparison came out six days after OpenAI cut API and credit pricing for GPT-5.6 Sol by over 20%, a change dated August 21.
“Shouldn’t burn a hole in your pocket”
Simular co-founder and CTO Jiachen Yang told The New Stack that agents built for routine but necessary work, such as recruitment outreach, invoice validation and researching the latest news, “shouldn’t burn a hole in your pocket” just because the “underlying model was trained to solve the planet’s great unsolved open math conjectures.”
“Posterity will find it ludicrous that people are still building models that way right now,” Yang said. “Sai’s optimal cost-outcome tradeoff on OSWorld 2.0 is one step toward a future where we don’t need to settle for exorbitant prices to get the job done.”
Simular is based in Palo Alto and calls itself a “research-focused agentic startup.” Two ex-DeepMind scientists founded it: CEO Ang Li and Yang. It raised a $21.5 million Series A led by Felicis, with NVentures (Nvidia’s venture arm) and its seed investor South Park Commons also taking part. Simular first released Sai in March of this year. The agent works in full desktop applications and webpages, calls APIs and writes code, and the company promises “an accessible cost for individuals” and businesses. Simular says it works on economically valuable tasks, not demos, and that this requires owning the whole stack of a computer agent: the models, planning and grounding (matching an instruction to the right element on screen), the scripting layer, the virtual machines and the user interface.
Demos and production
Machine learning engineer Santiago Valdarrama pointed to the desktop on LinkedIn: “Because Sai operates on a full desktop, not just a browser or an API, it can handle applications that block standard automation. That opens up a whole class of tasks that most agents can’t touch.”
Replies in the same thread were more cautious. Baljinder Lally, who builds AI revenue systems, said Sai’s arrival “matches what I’ve seen building agent workflows,” then added: “The demos look great, but reliability comes from guardrails, visibility, and human-in-the-loop checkpoints.”
Jahanzaib A., founder of AgenticMode AI, said he had hit the same problem with voice agents. “They’d nail every demo then fail silently in production on edge cases nobody predicted. The observability part is everything – without that you’re just debugging blind.”
Up next
Claudeforce puts 37 Salesforce sales skills inside Claude
“Probabilistic intelligence alone doesn’t run a company,” Marc Benioff said of Claudeforce, Salesforce’s tie-up with Anthropic. It launches with a Claude plugin of 37 prebuilt sales skills, in pilot now and open beta in September.
Related stories
Comments
No comments yet. Be the first.
Join the conversation
Sign in to leave a comment. It appears after moderation.

