OpenAI’s Habitat cut read latency from 120 ms to 10 ms

Median reads from Habitat, OpenAI’s storage platform for ChatGPT and its other products, have dropped from roughly 120 ms to roughly 10 ms since the company started working on performance earlier this year. OpenAI says most of the 12x gain came from a cache placed near users and invalidated within seconds by the Azure Cosmos DB change feed. The details are in Part II of OpenAI’s engineering series, published October 23.

Habitat stores over 500 petabytes for more than 1 billion users in 40 geographic regions, and reads make up over 90% of its requests. The first part showed how the platform grew from a Python library into a service. This part covers what sits underneath: Azure Cosmos DB, the three tries it took to share it across products, and a working relationship with the Azure team that goes well past a normal support contract.

Why reads were slow

Compute and storage were often in different regions, so some requests had to cross regions no matter what. Product teams worked around the latency with client-side caching, request batching, fewer reads and writes, and moving work off the critical path. OpenAI counts that as a failure of Habitat’s own goal, which was to let teams ship fast without having to understand the database.

The fix comes in layers. The first is storing data near users, which works mainly where data residency rules require it. Elsewhere physical capacity gets in the way: many regions can’t hold the storage their share of traffic would need. Where datacenter space, power or cooling are tight, OpenAI has to choose between colocated storage and more serving compute.

The second layer is heavy replication, kept to small datasets that need the lowest latency, such as the authentication metadata every request touches. Some of these accounts replicate to all 40 Azure regions where OpenAI serves traffic, and others to at least one region per continent. Moving datasets between accounts with different replication setups meant building a transparent data movement and routing layer, earlier than the team wanted.

The cache covers everything else. Habitat stores recently accessed objects and edges in Valkey servers near where user requests are processed. A common cache design picks a TTL (how long an entry lives) and accepts the tradeoff: short TTLs mean more misses, long ones mean stale data. Invalidating on every write keeps data fresher, but partial failures leave stale entries behind. Across regions, the cache can also race the database’s own replication. Following Meta’s TAO, Habitat drives invalidation from change data capture, a stream of every write to the database. A service reads the Cosmos DB change feed and clears the matching local cache entries everywhere within seconds. The next read misses and pulls the current version.

The last piece was a rewrite of the Habitat service itself from Python to Rust, which cut proxy overhead. The tail is still open. The slowest requests still fetch uncached data, often from distant regions, and OpenAI wants to place data closer to its readers from the start once more capacity comes online.

One container, many, one again

Habitat started with a single default Cosmos DB container per account. It relied on the documented limit of 10K RU/s per logical partition key (RU is Cosmos DB’s unit of throughput) to slow down clients that sent too much load. That broke in three ways. Clients picked their own partition keys, so common ones like user_id and workspace_id collided across products and shared one limit. The limit is actually enforced at the physical partition, so traffic for a niche feature could disrupt login and signup. And because a container’s throughput is spread evenly across its physical partitions, storage growth caused splits, and each child partition got half its parent’s throughput. Storage growth on its own cut capacity, even with request rates flat.

Next came dedicated containers for critical use cases, plus a script that kept each container’s throughput in line with its partition count. That limited outages to single features rather than the whole site. But a new feature might run at 1K RU/s during load testing and hit 1M RU/s at launch. Teams forgot to pre-scale, and the storage team spent launch days in SEV rooms trying to scale containers at the last minute. In practice, nearly every product team at OpenAI ended up doing its own capacity planning, some more carefully than others.

The third phase went back to one shared container, now with protections built together with Azure. Hot-key throttling slows the hottest keys first. Request tiers put offline work, such as bulk inserts and data extracts, behind interactive traffic. The existing serverless mode would have fixed weak partitions, but it scaled only on storage and capped each physical partition at 5K RU/s. The Azure Cosmos DB team built OpenAI a new serverless mode that lets each physical partition reach 10K RU/s and also splits partitions on RU usage, not only on storage. OpenAI reports no provisioning or noisy-neighbor incidents in the months since, so the fix for a widely shared multi-tenancy problem turned out to be a database mode written for a single customer.

“Less managed” at scale

OpenAI’s largest Cosmos DB account holds tens of petabytes and serves more than 20M QPS, and the company believes no one else runs Cosmos DB at that size. Two incidents showed what that scale means. In late 2024, a power outage hit the primary region of one of its largest accounts, and failing over to a secondary region took several hours, a scenario neither team had tested at that size. About a month later, failing back caused a second serious outage. A quirk in the gateway-mode client sent every client to the original region at once, though only some physical partitions had moved back, and the resulting connection stampede overwhelmed the region’s gateway.

“At scale, managed services become less managed,” the team writes. Today Habitat and Azure engineers monitor every layer together, from OpenAI’s services down to Cosmos DB’s physical storage. They test changes on a production-sized account that serves no production traffic, work on the SDKs together, and can page each other directly without going through support tiers. Over the past year, joint work added load protection and metadata caching to the Cosmos DB gateway. OpenAI cut connection pressure from its Python fleet with pooling and reuse, and moved offline workloads into separate VPCs from critical ones.

The change-data-capture pipeline behind the cache runs at a similar scale: millions of changes per second, from thousands of containers, through more than a hundred Kafka topics. The same pipeline feeds secondary indexes, data movement and analytics. OpenAI even changed Azure’s CDC SDK so it batches checkpoints in memory and writes them out periodically instead of one at a time.

For its next billion users, OpenAI is building its own exabyte-scale online store for bulky Habitat data. It combines the RocksDB Cloud core from Rockset with Raft-based replication and uses the same document model as Cosmos DB. It separates storage from compute and replicates at the physical level to need less flash capacity, and in exchange it supports a lower peak QPS.