Two sample agents run code to drive browser and desktop

OpenAI has published a sample app on GitHub with two computer-use agents that drive software by writing code: JavaScript with Playwright for a browser, Python with PyAutoGUI for a desktop. Both run on the Responses API and share one web console and one set of lab environments.

At a glance

• A persistent runtime keeps state and helper functions available between model calls, so the agent can loop through several fields, verify each change took effect and return only the text or screenshots it needs.
• The repository ships three lab templates, a kanban board, a drawing canvas and a hotel booking mock, and pins Node.js 22.20.0, pnpm 10.26.0 and Python 3.12 for local runs.
• OpenAI states the samples carry no operating-system sandbox and none of its production action-review controls: generated code runs with the user's permissions, and the Python agent moves the real mouse and keyboard.

Publishing the loop itself rather than a product reads as an attempt to set a reference shape for computer-use agents: the expensive part of these systems is the round trips, and code that batches actions and filters observations attacks exactly that cost. The safety notes also mark the boundary of the release, since anything running with full user permissions on a real desktop likely stays confined to controlled machines.

The runtime keeps state between calls so the model can batch actions

Computer-use agents inspect an interface, choose an action, execute it and check the result. OpenAI builds that loop around models that write code, so a single call can combine several actions, process the observations that come back and decide when to look again. OpenAI says this reduces model round trips and repeated input context while giving the model feedback to correct mistakes.

The loop lives in responses-loop.ts on the JavaScript side and responses_loop.py on the Python side, with persistent execution workers in javascript-worker.ts and app/desktop/worker.py, and run lifecycle handled by runner-manager.ts and runner.py. OpenAI states the samples show how to build the pattern with standard libraries, while its own products use their own runtimes and additional controls.

JavaScript with Playwright drives the browser, Python with PyAutoGUI drives the desktop

The JavaScript agent uses Playwright locators, screenshots and browser controls in a persistent session, with a TypeScript server and agent loop. The Python agent runs its server, agent loop and a persistent PyAutoGUI worker in Python, controlling a visible browser on the desktop through screenshots, mouse input and keystrokes.

Setup pins versions: Node.js 22.20.0 with Corepack using the repository's pnpm 10.26.0, and, for the Python app, uv with Python 3.10 or newer, pinned at 3.12 in the repository. Both apps read the root .env file, and shell variables take precedence over it.

Once an app starts, the shared console lists scenarios, opens their prompts for editing and starts a run; runs make real API calls. Stop interrupts a run, and OpenAI advises Ctrl+C and a completed shutdown before switching between the two apps.

Each run copies a lab template and records a replay trace

The repository ships three lab templates: a kanban board with cards, columns and tasks; a paint canvas with shapes, layers and tools; and a booking lab with hotel search and mock reservations. Scenario defaults and task prompts live in labs/catalog.json, shared by both agents.

Every run gets a fresh copy of a template. Workspaces, screenshots and replays stay in the selected app's ignored data/ directory, and the console exposes a Replay JSON trace and a screenshot timeline for reviewing earlier states of the environment.

In the documentation, a finished run means only that the agent loop and cleanup ended normally: OpenAI states a final answer does not prove the task succeeded, and the screenshots, recorded trace and model response have to be inspected. The code is released under the MIT License.

No sandbox in the samples

Generated code runs with the user's permissions, and OpenAI states the samples provide neither an operating-system sandbox nor the production action-review controls it uses in its own products. The Python agent moves the real mouse and keyboard, and screenshots may capture other windows.

OpenAI recommends a dedicated desktop session with the lab window kept in front, services on default loopback addresses, and API keys and run artifacts kept private. A crash can leave desktop input held down, a case covered by the Python app's interruption and recovery notes.