OpenAI’s Habitat grew from a Python library to 22M req/sec

Habitat, the storage layer behind ChatGPT, started out as a Python library that product teams imported into their own code. It is now a standalone, globally distributed service handling 22M requests per second for 1 billion users, OpenAI wrote in a September 11 post on the OpenAI engineering blog. The request rate is OpenAI’s own figure. The user count matches ChatGPT’s scale: the service passed 1 billion monthly active users by June.

The library model ran out of road by mid-2025. OpenAI was moving to regionally distributed Cosmos DB accounts, and every change had to ship inside the client used by dozens of services. Rolling out a new client took days, adding shadowing took more days, and fixing bugs took more again. Partway through that rollout, one team rolled back to an old, buggy client version for unrelated reasons. That one rollback was enough to cause exactly the kind of incident the whole process was meant to prevent.

So OpenAI pulled Habitat out into its own service. Client services now talk to Habitat, and Habitat talks to storage resources such as Azure Cosmos DB. Clients only see Habitat’s simple API and never need to know how the data is spread underneath. Deployments, observability and fixes now go through a single control point. Habitat is also a security chokepoint: access control lists, audit logging and storage access restrictions are enforced in one place, and that includes requests from agentic actors.

Complex queries don’t touch the online path at all. OpenAI keeps an offline secondary view of Habitat in Rockset, fed by change data capture (a stream of every write to the primary store). Each team runs its own instance, which keeps read-heavy analytics and search away from the storage that serves live traffic.

The service itself was written in Python, and that is where it hit its next limit. asyncio gives concurrency but not CPU parallelism, because of the GIL. Habitat does a lot of CPU-heavy work on every request: routing, compression, encryption, checksums, health checks, shadowing and hedging. Traces of the slowest requests, at p99 and above, showed the downstream database answering quickly while the coroutine sat waiting to be scheduled again. To catch this, OpenAI measured event loop delay with a background task scheduled at regular intervals.